Possibly you're using technical jargon here. When non-LessWrong-reading humans talk about one person imposing their values on everyone else, they would generally consider it immoral. Are we in agreement here?

Now, I could understand your starement ("No it's not") in either of two ways: Either you believe they're mistaken about whether the action is immoral, or you are using a different (technical jargon) sense of the words involved. Which is it?

My guess is that you're using a technical sense of "values", which includes something like the various clauses enumerated in EY's description of CEV: "volition is what we wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together, ...".

If by "values" you include those things that you don't think you value now but you would value if you had more knowledge of them, or would be persuaded to value by a peer if you hadn't conquered the world and therefore eliminated all of your peers, then perhaps I can see what you're trying to say.

By talking about "imposing your own values" without all of the additional extrapolated volition clauses, you're committing an error of moral overconfidence - something which has caused vast amounts of unpleasantness throughout human history.

http://www.overcomingbias.com/2009/01/moral-uncertainty-towards-a-solution.html