That's helpful. But it also seems unduly restrictive. I realize that you're not saying that we literally have to treat our own minds as immaterial entities (are you?), but it still seems a pretty high price to pay. Can I treat the epistemic states of my loved ones as part of the outcome? Presumably so, so why can't I give myself the same consideration? I'm trying to make you feel the cost, here, as I see it.
Hm. I haven't thought much about that. Maybe there is something interesting to be said about what aspects of an agent's internal state can they have preferences over for there still to be an interesting rationality theorem? If you let agents have preferences over all decisions, then there is no rationality theorem.
I don't believe the VNM theorem describes humans, but on the other hand I don't think humans should endorse violations of the Independence Axiom.
Followup to : Is risk aversion really irrational?
After reading the decision theory FAQ and re-reading The Allais Paradox I realized I still don't accept the VNM axioms, especially the independence one, and I started thinking about what my true rejection could be. And then I realized I already somewhat explained it here, in my Is risk aversion really irrational? article, but it didn't make it obvious in the article how it relates to VNM - it wasn't obvious to me at that time.
Here is the core idea: information has value. Uncertainty therefore has a cost. And that cost is not linear to uncertainty.
Let's take a first example: A is being offered a trip to Ecuador, B is being offered a great new laptop and C is being offered a trip to Iceland. My own preference is: A > B > C. I love Ecuador - it's a fantastic country. But I prefer a laptop over a trip to Iceland, because I'm not fond of cold weather (well, actually Iceland is pretty cool too, but let's assume for the sake of the article that A > B > C is my preference).
But now, I'm offered D = (50% chance of A, 50% chance of B) or E = (50% chance of A, 50% chance of C). The VNM independence principle says I should prefer D > E. But doing so, it forgets the cost of information/uncertainty. By choosing E, I'm sure I'll be offered a trip - I don't know where, but I know I'll be offered a trip, not a laptop. By choosing D, I'm no idea on the nature of the present. I've much less information on my future - and that lack of information has a cost. If I know I'll be offered a trip, I can already ask for days off at work, I can go buy a backpack, I can start doing the paperwork to get my passport. And if I know I won't be offered a laptop, I may decide to buy one, maybe not as great as one I would have been offered, but I can still buy one. But if I chose D, I've much less information about my future, and I can't optimize it as much.
The same goes for the Allais paradox: having certitude of receiving a significant amount of money ($24 000) has a value, which is present in choice 1A, but not in all others (1B, 2A, 2B).
And I don't see why a "rational agent" should neglect the value of this information, as the VNM axioms imply. Any thought about that?