So that when someone says they're going 5 miles per hour, you don't think "they'll have gone 5 miles after one hour" but rather "1/5 of an hour will have passed after they've gone one mile"?

I don't think any of those things. I wonder about how far they're going at that speed, and if the answer is "up the block" I think "oh, they'll be there soon" and if the answer is "to the moon" I think "that's going to take forever". I do not naturally think in numbers.

And if I wanted you to think "they'll have gone 5 miles after one hour", I would have to say "they're going 1/5 hours per mile"?

No, if you want me to think that they will have gone five miles after an hour, you tell me they're going someplace five miles away and it'll take them an hour to get there.

I thought you were talking about thinking in terms of a two-dimensional map with different routes to the same place having different distances.

Well, I was, but this was incidental to my point.

I suppose in retrospect I can make sense of it by interpreting the phrase "specific destination" as referring to keeping the distance fixed (and letting time vary).

Yes.

you shouldn't have been confused by the original problem at all

I was not confused in the way you were. I was confused in a different way, which has nothing to do with how I read the English word "speed" and everything to do with how my brain generates error messages when presented with math problems.

"If you want to average 40 mph on a two-hour trip, and went 20 mph for the first hour, what should your speed be for the second hour?"

Well, this is also confusing (my brain generates "sixty" automatically, but I don't actually know if that's an answer to this problem or just the result of seeing "40", "20", and "average" in that order, and I would have to do work to find out). It is not differently confusing than the first problem. I don't get any farther or stop any earlier before I want to seek assistance. (I don't even know if this is the same problem or not.)