I guess this all depends on how several things come together ...
... is here a notion of "responsible use of property" or is there a kind of absolutist propertarianism - i.e. if you own something, that comes with no customary or legal responsibility toward society to put it to use in a 'proper' and 'productive' way that contributes collectively.
... is slavery always unconditional or is it sometimes conditional (as in the examples of debt slaves who are enslaved for a set period of time)
... and what is the default state of a person, if a master has been found to fail at his responsibilities of ownership, are there situations where the slave defaults to freedom, or does one say 'this slave must be owned by someone'
The judicial system interfering with, and punishing 'bad' slave owners might be something that the majority of owners who see themselves as 'well-behaved' wouldn't oppose ... the bad owners would be seen as shirking responsibilities that the good ones have to shoulder, and/or reaping short term profits that the good ones have to forego.
This would be less based on any idea of punishing masters for violating 'equal human rights' of slaves, but rather punishing them for failing to fulfill their duty to the greater society. And by definition a master's misbehavior would have to go far beyond 'the usual'. But I don't see that 'annulling slave ownership' couldn't be a possible sanction used against irresponsible masters in a slave-owning society.