I'm still bothered by the fact that different people mean different and in fact contradictory things by "moral realism".

This is a strong argument against moral realism. If the thing were true, it would be easier to define - or at least, different people's definitions would be of the same object, even if they explained it differently.