I suspect it's really counterintuitive to most people. That's why it gets so much discussion, and in particular why so many people fight the hypothetical so hard. The "yeah, that makes sense, but then my brain starts screaming" reaction is pretty common.

And yes, I agree that if we compare things that are closer together in scale, our intuitions don't break down quite so dramatically.