At any level of technology.
I am not going to believe that. Both because at the caveman level the fossil fuels are pretty much useless and because your imagination with respect to future technology seems severely limited.
"Unit of infrastructure" ~= amount of energy and effort and capital needed to get at it.
This entirely depends on the technology level. And how are you applying concepts like "energy-dense" to, say, sunlight or geothermal?
at the caveman level the fossil fuels are pretty much useless
Coal was used as fuel before the Roman empire. It didn't lead to an industrial revolution until someone figured out a way to turn it into mechanical energy substituting for human labor instead of just a heat source in a society where that could be made profitable due to a scarcity of labor. That was the easiest, surface-exposed deposits, yes, but you hardly need any infrastructure at all to extract the energy, and even mechanical energy extraction just needs a boiler and some pistons and val...
You know the drill - If it's worth saying, but not worth its own post (even in Discussion), then it goes here.