Just to clarify, I am not the author of the original article; I haven't proposed any particular naive decision theory. (I don't think "treat indexicals as suspect" qualifies as one!)
Unfortunately, my mind is not the product of a clearthinking AI and my values do (for good or ill) have indexically-defined stuff in them: I care more about myself than about random other people, for instance. That may or may not be coherent, but it's how I am. And so I don't know what the "right" way to get indexicals out of the rest of my thinking would be. And so I'm not sure how I "should" bet in the presumptuous philosopher case. (I take it the situation you have in mind is that someone offers me to bet me at 10:1 odds that we're in the Few People scenario rather than the Many People scenario, or something of the kind.) But, for what it's worth, I think I take the P.P.'s side, while darkly suspecting I'm making a serious mistake :-). I repeat that my thinking on this stuff is all much less than half-baked.
I am not the author of the original article
I realized this, but I think my mind attached some of snarles' statements to you.
I agree with your choice in the presumptuous philosopher problem, but I doubt that anyone could actually be in such an epistemic state, basically because the Few People cannot be sure that there is not another universe with completely different laws of physics simulating many copies of theirs, as well as many other qualitatively similar possible scenarios.
I am posting this is because I'm interested in self-modifying agent decision theory but I'm too lazy to read up on existing posts. I want to see a concise justification as to why a sophisticated decision theory would be needed for the implementation of an AGI. So I'll present a 'naive' decision theory, and I want to know why it is unsatisfactory.
The one condition in the naive decision theory is that the decision-maker is the only agent in the universe who is capable of self-modification. This will probably suffice for production of the first Artificial General Intelligence (since humans aren't actually all that good at self-modification.)
Suppose that our AGI has a probability model for predicting the 'state of the universe in time T (e.g. T= 10 billion years)' conditional on what it knows, and conditional on one decision it has to make. This one decision is how should it rewrite its code at time zero. We suppose it can rewrite its code instantly, and the code is limited to X bytes. So the AGI has to maximize utility at time T over all programs with X bytes. Supposing it can simulate its utility at the 'end state of the universe' conditional on which program it chooses, why can't it just choose the program with the highest utility? Implicit in our set-up is that the program it chooses may (and very likely) will have the capacity to self-modify again, but we're assuming that our AGI's probability model accounts for when and how it is likely to self-modify. Difficulties with infinite recursion loops should be avoidable if our AGI backtracks from the end of time.
Of course our AGI will need a probability model for predicting what a program for its behavior will do without having to simulate or even completely specify the program. To me, that seems like the hard part. If this is possible, I don't see why it's necessary to develop a specific theory for dealing with convoluted Newcomb-like problems, since the above seems to take care of those issues automatically.