It said something about a person that he tried not to bother house elves. Specifically, it said that he'd been Sorted into Hufflepuff, since, to the best of Harry's knowledge, Hermione was the only non-Hufflepuff who worried about bothering house elves. (Harry himself thought her qualms rather silly. Whoever had created house elves in the first place had been unspeakably evil, obviously; but that didn't mean Hermione was doing the right thing now by denying sentient beings the drudgery they had been shaped to enjoy.)

Chaper 42.