That's interesting though - are there any fields are that suck purely out of necessity? What's the mechanism that causes it?

I think the perspective to take would be one of 'market failures'. Either there are large improvements to be had in urban planning, or there are not. If they are not, then the whole discussion is pointless, so let's assume there are. If there are large improvements to be had, then by Coase's theorem in a functioning market, they will get made. They are not getting made. Therefore, urban planning is not a functioning 'efficient market' in any sense. Why? Rent-seeking (literally), conflicting interests, politics, expense of construction, legacy costs of infrastructure (see any major construction in NYC or Boston compared to, say, Shanghai or Beijing), etc. I don't know, ask a specialist why the dreams of urban planners so often die.