I'm nervous about the idea that life might adapt to conditions in which it cannot originate. Unless you mean spores, but they have to wait for the world to warm up.
As for panspermia, we have a few billion years of modern conditions before the Earth, which is itself already a problem. I think the natural comparison is the size of that Goldilocks zone to the very early one. But I don't know which is bigger.
Here are three environments. Which is better for radiation of spores?
(1) a few million years where every planet is wet
(2) many billion years, all planets cold
(3) a few billion years, a few good planets.
The first sounds just too short for anything to get anywhere, but the universe is smaller. If one source of life produces enough spores to hit everything, then greater time depth is better, but if they need to reproduce along the way, the modern era seems best.
I'm nervous about the idea that life might adapt to conditions in which it cannot originate.
Why this happened on Earth? It is pretty likely for example that life couldn't originate in an environment like the Sahara desert, but life can adapt and survive there.
I do agree that spores are one of the more plausible scenarios. I don't know enough to really answer the question, and I'm not sure that anyone does, but your intuition sounds plausible.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.