The specific situation you describe also can't possibly arise, because one would look down at me to speak to me and the other would look up. Then they'd be behaving differently.

ETA: I tried to think of a least convenient world, but couldn't.

Suppose you're standing on a staircase. The taller man stands on a step below you, while the shorter man stands on a step above you, and the steps are of such height that each would be looking you directly in the eye. Is that a sufficiently inconvenient world?