Of course, but it still has an effect. And also the tall guy standing a step below me is definitely not behaving the same as the short guy standing a step above me.

Huh, yeah. He's also wearing larger clothes, and curving spacetime by a larger amount. But “all other things” in “all other things being equal” doesn't usually literally mean all other things -- otherwise any counterfactual will involve logical inconsistencies.