If agents have different utility functions / conative ambitions / "shoulds", they will presumably need to engage in some kind of negotiation in order to compromise their values and reach an efficient outcome. Presumably, ethical disputes can function as a way of reaching such outcomes - some accounts of ethics are quite clear in describing ethical reasoning as being very much about such a balancing of "right versus right". Even Kantian ethics can be seen in such terms, although what we woud call "rights" Kant would perhaps refer to as "principles of practical reason".

One could try to create a model of agents that respond to such exhortations. Maybe such agents could be uncertain about their own utility function, like in Dewey's value learning paper.