Let us say you are playing Steve Omohundro's meal choosing game 1, however the negatives are a bit harsher and more realistic than just a dodgy soufle. You are given two choices on the menu, oysters and fugu. Your goal avoid death, sickness and eat tasty food. You don't know much about either, although you do know that shellfish has made you ill in the past so you give it a lower expected utility (pretend you don't know what fugu is).
Eating the poorly prepared fugu kills you dead every time, do not pass go, do not update your utility values of choosing an option (although the utility of it would be 0, if you were allowed to update). Eating oysters gives you a utility of 1.
So how do we win in this situation? In a way it is easy: Don't eat the fugu! But by what principled fashion should you choose not to eat the fugu? Microeconomics is not enough, with negative expected utility from shellfish you would pick the fugu! Also you do not get to update your utilities when you eat the fugu, so your expected utilities can't converge with experience. So we are in a bit of a pickle.
Can humans solve these kinds of problems, if so how do we do it? The answer is poorly, in a patch work fashion and we get information on the fugu type problems from our genome and culture. For example we avoid bitter things, are scared of snakes and are careful if we are up high are because our ancestors had to have had thes bits of information (and more) to avoid death. They got them by chance, which isn't exactly principled. But all these are still needed for winning. We can also get the information culturally, but that can leave us open to taboos against harmless things such as eating pork, which we might be foolish to test ourselves. It is hardly principled either.
So in this kind of scenario it is not sufficient to be economically rational to win, you have to have a decent source of knowledge. Getting a decent source of knowledge is hard.
1 See the appendix of the Nature of Self-Improving Artificial Intelligence starting page 37
Whpearson----I think I do see some powerful points in your post that aren't getting fully appreciated by the comments so far. It looks to me like you're constructing a situation in which rationality won't help. I think such situations necessarily exist in the realm of platonic possibility. In other words, it appears you provably cannot always win across all possible math structures; that is, I think your observation can be considered one instance of a no free lunch theorem.
My advice to you is that No Free Lunch is a fact and thus you must deal with it. You can't win in all worlds, but maybe you can win in the world you're in (assuming it's not specially designed to thwart your efforts; in which case, you're screwed). So just because rationality has limits, does not mean you shouldn't still try to be rational. (Though also note I haven't proven that one should be rational by any of the above).
Eli addressed this dilemma you're mentioning in passing the recursive buck and elsewhere on overcoming bias)
My point is slightly different from NFL theorems. They say if you exhaustively search a problem then there are problems for the way you search that mean you will find the optimum last.
I'm trying to say there are problems where exhaustive search is something you don't want to do. E.g. seeing what happens when you stick a knife into your heart or jumping into a bonfire. These problems also exist in real life, where as the NFL problems are harder to make the case that they exist in real life for any specific agent.