I wonder about how far they're going at that speed, and if the answer is "up the block" I think "oh, they'll be there soon" and if the answer is "to the moon" I think "that's going to take forever". I do not naturally think in numbers.

Well, neither do I (I naturally think in terms of operations and transformations), so that's not the relevant distinction. The relevant distinction is between "wow, they're already far away" (speed) vs. "wow, they got there quickly" (inverse speed).

Let me see if I can generate, in your mind, something analogous to the confusion that existed in my mind. I probably won't succeed, but the idea of attempting is too interesting to resist.

Here are two questions that are easy to answer:

(1) If I travel for an hour and spend a lot of time on the first part of my journey, how much time will I spend on the second part? (Answer: not much.)

(2) If I travel a mile and go a large distance during the first part of my journey, how far will I have to go during the second part? (Answer: not very far)

And now here are two questions that are confusing:

(3) If I travel for an hour and go a large distance during the first part of the journey, how much time will I have to spend on the second part? (Answer: Huh? That depends on how much time you spent going that large distance during the first part.)

(4) If I travel a mile and spend a long time on the first part of the journey, how much distance will I have to cover on the second part? (Answer: Huh? That depends on how much distance you covered during that long time you spent on the first part.)