One of the previous comments (I think in the previous post) pointed out that yes, indeed this does occur - but that these effects mainly cancel each other out.

In my mind this works somewhat like Brownian motion: lots of tiny pushes, but overall, it continues in roughly the same way. ie, mostly the photon carries on as though it hasn't changed configuration space in any significant way.