Though there's logic to this argument, pretty much everything else about the way house elves were made is evil. They're created, or conditioned to brutally torture themselves if they even think they've displeased their masters or broken a rule. They have no labor rights and can be mistreated at will, to the point that mistreatment is built in as a product feature.

We can only imagine what sort of miserable Dickensian conditions they live in when they're not at work. They're forced to wear ragged, salvaged sacks, as giving them clothes = firing them, i.e. denying them the work and subservient position they're designed to want. This is a needless cruelty on top of everything else. Heck, if I were an aristocrat wizard with house elves, I'd want mine to go around in elegant livery, as a demonstration of how magnificent my Estate is. But I couldn't do that, because the poor little creatures were made (modified?) by a sadist.