Take a second to go upvote You Are A Brain if you haven't already...
Back? OK.
Liron's post reminded me of something that I meant to say a while ago. In the course of giving literally hundreds of job interviews to extremely high-powered technical undergraduates over the last five years, one thing has become painfully clear to me: even very smart and accomplished and mathy people know nothing about rationality.
For instance, reasoning by expected utility, which you probably consider too basic to mention, is something they absolutely fall flat on. Ask them why they choose as they do in simple gambles involving risk, and they stutter and mutter and fail. Even the Econ majors. Even--perhaps especially--the Putnam winners.
Of those who have learned about heuristics and biases, a nontrivial minority have gotten confused to the point that they offer Kahneman and Tversky's research as justifying their exhibition of a bias!
So foundational explanatory work like Liron's is really pivotal. As I've touched on before, I think there's a huge amount to be done in organizing this material and making it approachable for people that don't have the basics. Who's going to write the Intuitive Explanation of Utility Theory?
Meanwhile, I need to brush up on my Python and find a way to upvote Liron more than once. If only...
Update: Tweaked language per suggestion, added Kahneman and Tversky link.
I do not think your claim is what you think it is.
I think your claim is that some people mistake the model for the reality, the map for the territory. Of course models are simpler than reality! That's why they're called "models."
Physics seems to have gotten wiser about this. The Newtonians, and later the Copenhagenites, did fall quite hard for this trap (though the Newtonians can be forgiven to some degree!). More recently, however, the undisputed champion physical model, whose predictions hold to 987 digits of accuracy (not really), has the humble name "The Standard Model," and it's clear that no one thinks it's the ultimate true nature of reality.
Can you give specific examples of people making big mistakes from map/territory confusion? The closest thing I can think of offhand is the Stern Report, which tries to make economic calculations a century from now based on our current best climate+social+political+economic models.