So why is our world so orderly? There's a mathematically possible continuation of the world that you seem to be living in, where purple pumpkins are about to start falling from the sky. Or the light we observe coming in from outside our galaxy is suddenly replaced by white noise. Why don't you remember ever seeing anything as obviously disorderly as that?

Who says all of this is mathematically possible? I've read this idea before, and I think it's wrong.

First of all, I think it's very difficult to guess what is mathematically possible. We experience the universe at a level which is already extremely evolved. For example, I imagine that mathematically possibility resulted in an incredibly complex structure that eventually mapped to the rules of physics (string theory maybe, but certainly eventually quantum mechanics and then atoms, etc). Then the universe we experience is just the manifestation of that physics.

Secondly, another way to look at it, is that counter factual 'possibility' is not the same thing as mathematical possibility. For example I could have chosen not to compose this comment (counter-factually) but it wasn't actually possible that I wouldn't because I'm computing a program which -- certainly at this scale -- is deterministic.