It's potentially scary not because of the time difference, but because of the quantity of habitable planets. It's understood that current conditions in the Universe make it so that only relatively few planets are in the habitable zone. But if the Universe was warm, then almost all planets would be in the habitable zone, making the likelihood of life that much higher.

As I said in my reply to JoshuaZ though, the complication is that rocky planets were probably much rarer than they are now.