(This is the first, and most newcomer-accessible, post in a planned sequence.)
Newcomb's Problem:
Joe walks out onto the square. As he walks, a majestic being flies by Joe's head with a box labeled "brain scanner", drops two boxes on the ground, and departs the scene. A passerby, known to be trustworthy, comes over and explains...
If Joe aims to get the most money, should Joe take one box or two?
What are we asking when we ask what Joe "should" do? It is common to cash out "should" claims as counterfactuals: "If Joe were to one-box, he would make more money". This method of translating "should" questions does seem to capture something of what we mean: we do seem to be asking how much money Joe can expect to make "if he one-boxes" vs. "if he two-boxes". The trouble with this translation, however, is that it is not clear what world "if Joe were to one-box" should refer to -- and, therefore, it is not clear how much money we should say Joe would make, "if he were to one-box". After all, Joe is a deterministic physical system; his current state (together with the state of his future self's past light-cone) fully determines what Joe's future action will be. There is no Physically Irreducible Moment of Choice, where this same Joe, with his own exact actual past, "can" go one way or the other.
To restate the situation more clearly: let us suppose that this Joe, standing here, is poised to two-box. In order to determine how much money Joe "would have made if he had one-boxed", let us say that we imagine reaching in, with a magical sort of world-surgery, and altering the world so that Joe one-boxes instead. We then watch to see how much money Joe receives, in this surgically altered world.
The question before us, then, is what sort of magical world-surgery to execute, before we watch to see how much money Joe "would have made if he had one-boxed". And the difficulty in Newcomb’s problem is that there is not one but two obvious world-surgeries to consider. First, we might surgically reach in, after Omega's departure, and alter Joe's box-taking only -- leaving Omega's prediction about Joe untouched. Under this sort of world-surgery, Joe will do better by two-boxing:
Expected value ( Joe's earnings if he two-boxes | some unchanged probability distribution on Omega's prediction ) >
Expected value ( Joe's earnings if he one-boxes | the same unchanged probability distribution on Omega's prediction ).
Second, we might surgically reach in, after Omega's departure, and simultaneously alter both Joe's box-taking and Omega's prediction concerning Joe's box-taking. (Equivalently, we might reach in before Omega's departure, and surgically alter the insides of Joe brain -- and, thereby, alter both Joe's behavior and Omega's prediction of Joe's behavior.) Under this sort of world-surgery, Joe will do better by one-boxing:
Expected value ( Joe's earnings if he one-boxes | Omega predicts Joe accurately) >
Expected value ( Joe's earnings if he two-boxes | Omega predicts Joe accurately).
The point: Newcomb's problem -- the problem of what Joe "should" do, to earn most money -- is the problem which type of world-surgery best cashes out the question "Should Joe take one box or two?". Disagreement about Newcomb's problem is disagreement about what sort of world-surgery we should consider, when we try to figure out what action Joe should take.
I freely admit that the problem may still be above my pay grade at this point, but your comment does accurately describe my dissatisfaction with some handlings of Newcomb's problem I've seen in rationalist circles. It's like they want the decision to have everything we recognize as "causing", but not call it that.
Perhaps it would help to repeat an analogy someone made a while back here (I think it was PhilGoetz). It's a mapping from Newcomb's problem to the issue of revenge:
Have the disposition to one-box --> Have the disposition to take revenge (and vice versa)
Omega predicts you'll one-box --> People deem you the type to take revenge (perhaps at great personal cost)
You look under the sealed box --> You find out how people treat you
You actually one-box --> You actually take revenge
The mapping isn't perfect -- people don't have Omega-like predictive powers -- but it's close enough, since people can do much better than chance.
What happens when I one-box and find nothing? Well, as is permitted in some versions, Omega made a rare mistake, and its model of me didn't show me one-boxing.
What happens when I'm revenge-oriented, but people cheat me on deals? Well, they guessed wrong, as could Omega. But you can see how the intention has causal influence, which ends once the "others" make their irreversible choice. Taking revenge doesn't undo those acts, but it may prevent future ones.
Apologies if I've missed a discussion which has beaten this issue to death, which I probably have. Indeed, that was the complaint when (I think) PhilGoetz brought it up.
Update: PhilGoetz was the one who gave me the idea, in what was a quite reviled top-level post. But interestingly enough, in that thread, Eliezer_Yudkowsky said that his decision theory would have him take revenge, and for the same reason that he would one-box!
And here's my remark showing my appreciation for PhilGoetz's insight at the time. And under-rated post on his part, I think...