Of course I would not be OK. I would want them to change their behavior and I would try to change it. This would be because of my preferences as to how people should behave towards me. These preferences don't exist independently of me. Morals are a special kind of preferences.

Saying "there exist someone's morals but there do not exist morals by themselves" is exactly the same as saying "there exist someone's preferences but there do not exist preferences by themselves".