It looks like he's formed his identity around that loyalty, and it might do a number on his head if he had to suddenly act in an inconsistent manner.

A fourth-year Slytherin who doesn't understand the concept of pretending in public isn't worth feeding to a pet snake. He doesn't have to speak against his mother so much as to keep quiet and find a way to avoid bullies.

But what you say is true. All the more reason he should associate with Hufflepuffs - they can appreciate loyalty.