What if the the answer to Fermi's paradox is simply "You'd have to mistake the infrastructural equivalent of a blood sugar rush for an inexorable trend in technological development to even wonder why nobody's zipping around in relativistic spacecraft or building Dyson spheres?" What if the problem is just short time horizons and poor understanding of context?

This. I've been searching for a way to articulate this idea for quite some time, and this is the best way I've seen it stated.

The last few centuries are potentially extremely atypical in human history. We have three generations of economists raised on exponentiation to think it is normal, and a series of technological advances that almost all require highly concentrated energy in ways that are seldom appreciated. When you think about it too, it would appear that something like an oil well is the most concentrated source of easily captured energy in the solar system - where else do you get such a huge amount of highly reduced matter next to such highly oxidized gas? With the interface between them requiring something as simple as a drill and furnace? Per unit of infrastructure and effort that is an incredible resource that I honestly doubt you can really improve upon. I have long suspected that reversion towards (through perhaps not all the way to) the mean is far more likely in our future.