This seems to me to be a permutation of the sunk cost fallacy. If you retrace your steps, you feel as though you're wasting the effort you took to get there in the first place, even though it's already spent and now it makes sense to go back again.
I share the intuition, by the way.
Picture a circular road on a map. Let's say that my office is at twelve o'clock, my home is at five o'clock, and the post office is at three o'clock.
Now, suppose I have to leave work, pick up a document at home, and take it to the post office to mail it. I know it's faster to walk clockwise home, passing the post office, and then return to it with the letter. But my gut preference is to go counterclockwise, either because of an aversion to retracing my steps, or because that route just ... feels "cleaner" or more efficient somehow, or ... I can't articulate it any better than that.
Does anyone else share this intuition? Is it a manifestation of one or more known/studied effects?