To me the part that stands out the most is the computation of P() by the AI.

This module comes in two versions: the module P, which is an idealised version, which has almost unlimited storage space and time with which to answer the question

From this description, it seems that P is described as essentially omniscient. It knows the locations and velocity of every particle in the universe, and it has unlimited computational power. Regardless of whether possessing and computing with such information is possible, the AI will model P as being literally omniscient. I see no reason that P could not hypothetically reverse the laws of physics and thus would always return 1 or 0 for any statement about reality.

Of course, you could add noise to the inputs to P, or put a strict limit on P's computational power, or model it as a hypothetical set of sensors which is very fine-grained but not omniscient, but this seems like another set of free variables in the model, in addition to lambda, which could completely undo the entire setup if any were set wrong, and there's no natural choice for any of them.

Jeremy, I think the apparent disagreement here is due to unclarity about what the point of my argument was. The point was not that this situation can't be analyzed with decision theory; it certainly can, and I did so. The point is that different decisions have to be made in two situations where the probabilities are the same.

Your discussion seems to equate "probability" with "utility", and the whole point of the example is that, in this case, they are not the same.

Jeremy, thank you for this. To be clear, I wasn't suggesting that meta-probability is the solution. It's a solution. I chose it because I plan to use this framework in later articles, where it will (I hope) be particularly illuminating.

I would take issue with the first section of this article in which you establish single probability (expected utility) calculations as insufficient for the problem.

I don't think it's correct to equate probability with expected utility, as you seem to do here. The probability of a payout is the same in the two situations. The point of this example is that the probability of a particular event does not determine the optimal strategy. Because utility is dependent on your strategy, that also differs.

This problem easily succumbs to standard expected value calculations if all actions are considered.

Yes, absolutely! I chose a particularly simple problem, in which the correct decision-theoretic analysis is obvious, in order to show that probability does not always determine optimal strategy. In this case, the optimal strategies are clear (except for the exact stopping condition), and clearly different, even though the probabilities are the same.

I'm using this as an introductory wedge example. I've opened a Pandora's Box: probability by itself is not a fully adequate account of rationality. Many odd things will leap and creep out of that box so long as we leave it open.

The substantive point here isn't about EU calculations per se. Running a full analysis of everything that might happen and doing an EU calculation on that basis is fine, and I don't think the OP disputes this.

The subtlety is about what numerical data can formally represent your full state of knowledge. The claim is that a mere probability of getting the $2 payout does not. It's the case that on the first use of a box, the probability of the payout given its colour is 0.45 regardless of the colour.

However, if you merely hold onto that probability, then if you put in a coin and so learn something about the boxes you can't update that probability to figure out what the probability of payout for the second attempt is. You need to go back and also remember whether the box is green or brown. The point of Jaynes and the A_p distribution is that it actually does screen off all other information. If you keep track of it you never need to worry about remembering the colour of the box, or the setup of the experiment. Just this "meta-distribution".

The exposition of meta-probability is well done, and shows an interesting way of examining and evaluating scenarios. However, I would take issue with the first section of this article in which you establish single probability (expected utility) calculations as insufficient for the problem, and present meta-probability as the solution.

In particular, you say

What’s interesting is that, when you have to decide whether or not to gamble your first coin, the probability is exactly the same in the two cases (p=0.45 of a $2 payout). However, the rational course of action is different. What’s up with that?

Here, a single probability value fails to capture everything you know about an uncertain event. And, it’s a case in which that failure matters.

I do not believe that this is a failure of applying a single probability to the situation, but merely calculating the probability wrongly, by ignoring future effects of your choice. I think this is most clearly illustrated by scaling the problem down to the case where you are handed a green box, and only two coins. In this simplified problem, we can clearly examine all possible strategies.

• Strategy 1 would be to hold on to your two dollar coins. There is a 100% chance of a $2.00 payout
• Strategy 2 would be to insert both of your coins into the box. There is a 50.5% chance of a $0.00 payout, 40.5% chance of a $4.00 payout and a 9% chance of a $2.00 payout.
• Strategy 3 would be to insert one coin, and then insert the second only if the first pays out. There is a 55% chance of $1.00 payout, a 4.5% chance of a $2.00 payout, and a 40.5% chance of a $4.00 payout.
• Strategy 4 would be to insert one coin, and then insert the second only if the first doesn't pay out. There is a 50.5% chance of a 0.00$ payout, a 4.5% chance of a $2.00 payout, and a 45% chance of a $3.00 payout.

When put in these terms, it seems quite obvious that your choice to open the box would depend on more than the expected payoff from only the first box, because quite clearly your choice to open the first box pays off (or doesn't pay off) when opening (or not opening) the other boxes as well. This seems like an error in calculating the payoff matrix rather than a flaw with the technique of single probability values itself. It ignores the fact that opening the first box not only pays you off immediately, but also pays you off in the future by giving you information about the other boxes.

This problem easily succumbs to standard expected value calculations if all actions are considered. The steps remain the same as always:

1. Assign a utility to each dollar amount outcome
2. Calculate the expected utility of all possible strategies
3. Choose the strategy with the highest expected utility

In the case of two coins, we were able to trivially calculate the outcomes of all possible strategies, but in larger instances of the problem, it might be advisable to use shortcuts in the calculations. However, it still remains true that the best choice will still be the one you would have gotten if you had done out the full expected value calculation.

I think the confusion arises because a lot of the time problems are presented in a way that screens them off from the rest of the world. For example, you are given a box, and it either has $10.00 or $100.00. Once you open the box, the only effect it has on you is the amount of money you got. After you get the money, the box does not matter to the rest of the world. Problems are presented this way so that it is easy to factor out the decisions and calculations you have to make from every other decision you have to make. However, decision are not necessarily this way (in fact in real life, very few decisions are). In the choice of inserting the first coin or not, this is simply not the case, despite having superficial similarities to standard "box" problems.

Although you clearly understand that the payoffs from the boxes are entangled, you only apply this knowledge in your informal approach to the problem. The failure to consider the full effects of your actions in opening the first box may be psychologically encouraged by the technique of "single probability calculations", but it is certainly not a failure of the technique itself to capture such situations.

she can conclude with >75% certainty that any male wants to fuck her.

... she can? Really? That seems pretty damn high for something as variable as taste in partners.

EDIT: wait, that's a reference to how many guys on a university campus will accept offers of one night stands, right? It's still too high, or too general.

Ah, I see. I think we were answering different questions. (I had this feeling earlier but couldn't pin down why.) I read the original question as being something like "what kind of hypotheses should a hypothetical AI hypothetically entertain" whereas I think you read the original question as being more like "what kind of hypotheses can you currently program an AI to entertain." Does this sound right?

Unless there are significant numbers of people, myself for example, who take the test multiple times with varied random algorithms just to see how it affects the outcome. I'd only put a (p=0.55) at the test underestimating your score, conditional that it doesn't correct for self-selection bias.

Though, given that the lowest score appears to be "less than 79", rather than an exact number, they may simply drop any scores under 79 from their pool, or at the very least weight them differently. Has anybody identified a similar maximum score which would support this hypothesis of discarding outliers?

Ah, I see. I think we were answering different questions. (I had this feeling earlier but couldn't pin down why.) I read the original question as being something like "what kind of hypotheses should a hypothetical AI hypothetically entertain" whereas I think you read the original question as being more like "what kind of hypotheses can you currently program an AI to entertain." Does this sound right?

Well, for starters, an AI living in a universe where infinitely many computations can be performed in finite time can verify the responses a Turing oracle gives it. So it can determine that it lives in a universe with Turing oracles (in fact it can itself be a Turing oracle), which is not what an AI living in this universe would determine (as far as I know).