"Physical" and "logical" are not the same thing. Even if all physical possibilities are instantiated (as Tegmark's Level IV Multiverse implies, I believe), there are logical systems that do not describe any part of reality.
I always say "physical/logical" to note the known laws of physics of our universe and the logic that describes it.
If you say only "physical" - then you limit yourselve only to that which is directly observable, testable and foreseeable. And that hinders a more relaxed approach of discussing such "far-out" possibilities as required in such cases.
Point being: IMO the only valid physical/logical speculations are those that relate to the physics and logic we know of (or a variation of it in an indeterministic universe),
Only Past ...