I think recent research into star formation rates in the universe actually might shed a lot of light on the situation and make some sense of our position in space and time. (My handle is CellBioGuy but I very nearly pulled off a double major in astronomy back in college, know a number of astronomy grad students, and follow astronomy closely.)

ARXIV, PDF Popular press

To make a long story short, a survery was made looking into deep space, and back in time up to 11 billion years, of a pretty good proxy of star formation (the emission lines produced by emission nebulae that are lit up like neon lights by the ultraviolet light of freshly-born huge stars). After doing some fancy math to correct for the expansion of space and the like the conclusions are striking - the modern average rate of star formation in the universe is less than 1/30 the peak rate of 11 billion years ago, and half of the stars in the universe are over 9 billion years old. To make matters even more interesting, the empirically-derived relationship between time and star formation actually converges to a finite number of stars when you project it into the future, and at infinity reaches a total number of stars born only 5% more than currently exist today.

95% of stars that will ever exist already exist.

This makes sense and has some interesting implications when you think in terms of galactic evolution and the history of the universe. Grand spirals like our galaxy are nearly the only places in the universe where star formation happens for billions of years on end and produces generations of stars rich in heavy elements. Dwarf galaxies go through one burst of star formation and the supernovas from the big stars blow all the gas out of their weak gravity, stopping star formation. Big elliptical galaxies form with nearly no angular momentum, so all their gas falls to the center, most of it becomes stars in one huge burst of low-metal stars, and the rest gets blown out of the galaxy when the central black hole gets activated. Grand spirals only wind down slowly with time as most of their gas stays away from the center due to angular momentum but their gravity makes sure that (almost) all the gas stays bound, getting more and more enriched in heavy elements with time.

...That is, until they collide with each other, which as most galaxies are part of clusters does eventually happen. It will happen to us in another four to five billion years, with andromeda. When this happens, the colliding galaxies go through one burst of star formation and then settle down into another elliptical galaxy. So over time, the number of star forming galaxies only decreases.

So, consider our position in space and time. We are in a grand spiral galaxy, which is the only sort of place that really produces high-metallicity stars. Our star system formed about 1/3 of the way through our galaxy's productive life before it collides with Andromeda (probably more like halfway through its compliment of stars, seeing as even spirals settle down with age), and we find ourselves currently about 2/3 of the way through its productive lifetime. I would call this an utterly typical position for an origin of life as we know it. Seeing how rapidly star formation is winding down across the universe as a whole, we do not find ourselves in an anomalously early point in time, even given that we are only 13 billion years into an apparently open-ended universe that will be around for at the very least trillions of years.

Our position becomes even more typical when you consider that Earth will only be habitable for at most another billion years before inevitable geochemical and astronomical processes turn us into another Venus, and that hilariously enough for a chunk of time before then the carbon content of our atmosphere will probably be so low that the photosynthetic production will be close to nill. We find ourselves evolving near the end of our planet's habitability window (consistent with complex life taking a while), in a system that formed about halfway through the window that produces clement systems.

This typical position only becomes a problem when you assume that intelligent systems either last longer than, say, a few million years, or they spread beyond their points of origin and consume the universe. The sheer number of ways that these two things could not be true lead me to think that they just don't happen, and that both our position in space and time and our nature are fairly typical of things that are smart enough to figure out where and when they are.

A lot of points here I find plausible. On the changing habitability of the universe, Earth has quite a lot of heavy elements. Those are, as I understand it, generally thought to be formed by supernovas, and so they should be rarer the further back you go in the history of the universe, as the earlier you go, the fewer supernovas there would have been so far. Some of the heavy elements are definitely involved in life (indeed, though I was mostly thinking of further up the periodic table, carbon and oxygen are relatively heavy compared to the majority of the matter in the universe, and not produced in any quantity by the normal activity of young stars, though they don't require supernovas). Further, just having enough heavier elements around may be important in planet formation.

Of course, this is all quite speculative. Perhaps some substantial amount of heavy elements was produced in the big bang. And very big stars have very short life cycles, so if there were a lot of very big stars when the universe was young, there might have been a lot of supernovas very early on. Or there might be other ways for heavy elements to form. But obviously if none of these were the case, and so heavy elements were much rarer in the early universe, that may mean that there are unlikely to be many star systems with remotely earthlike planets that are older than our solar system.

I think the Drake equation isn't really the way to estimate the probably of intelligent life - there is just too much we don't know to use the inside view. Take one thing like the Moon and we just don't know how important it was to have intelligent life on Earth (it definitely played a significant role, stabilizing the Earth orbit, and helping the sea => land transition with tidal forces) nor how rare such big satellites are.

To me the only way to look at it is to use the outside view, and do a reasoning like :

1. Look at us, and, like you said and realize it took us 5 billions of years to exist, which is actually half of the lifespan of Earth. We don't know if we were exceptionally fast or exceptionally slow, but it gives an order of magnitude of the time it takes for intelligent life to appear on a planet with good conditions.

2. Realize that it would be exceptional coincidence if another intelligent life would appear within 0.1% of the time it took us - so if there is intelligent life somewhere else, it's likely to be 5 millions of years in advance, or 5 millions of years late from us. 5 millions of years late means they are still apes.

3. Look around us, and see no trace of alien life - no Von Neumann probe visiting our system, no Dyson spheres or ringworlds around. And yet, if they were 5 millions of years in advance, we would very likely see such things.

So either there is a "great filter" and most of them didn't make it to space, or just, well, intelligent life is rare, because it takes in average 20 billions of years to appear even if conditions are good, and we were lucky to be fast enough to exist before the Sun became a red giant. Which that there will be occasional intelligent life in the universe, but so far apart that we didn't see them yet. If our nearest neighbor exists since 0.3 billion of years, but is 0.5 billions of light year from us, well, we'll sill need a long time before communicating with them.