I agree that a few million years is small, and that the low metal content would be a serious issue (which in addition to being a problem for life forming would also make planets rare as pointed out by bramflakes in their reply). However, the real concern as I see it is that if everything was like this for a few million years, then if life did arise (and you have a whole universe for it to arise), as the cooldown occurred, it seems highly plausible that some forms of life would have then adopted to the cooler environment. This makes panspermia more plausible and thus makes life in general more likely. Additionally, it makes more of a chance for life to get lucky if it managed to get into one of the surviving safe zones (e.g. something like the Mars-Earth biotransfer hypothesis).
I think you may be correct that this isn't a complete run around and panic level update, but it is still disturbing. My initial estimate for how bad this could be is likely overblown.
I'm nervous about the idea that life might adapt to conditions in which it cannot originate. Unless you mean spores, but they have to wait for the world to warm up.
As for panspermia, we have a few billion years of modern conditions before the Earth, which is itself already a problem. I think the natural comparison is the size of that Goldilocks zone to the very early one. But I don't know which is bigger.
Here are three environments. Which is better for radiation of spores?
(1) a few million years where every planet is wet
(2) many billion years, all planets ...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.