= 27d567bc2d48d9f40e9ccc20ea370b15)
incogn comments on Decision Theory FAQ - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (467)
I don't really think Newcomb's problem or any of its variations belong in here. Newcomb's problem is not a decision theory problem, the real difficulty is translating the underspecified English into a payoff matrix.
The ambiguity comes from the the combination of the two claims, (a) Omega being a perfect predictor and (b) the subject being allowed to choose after Omega has made its prediction. Either these two are inconsistent, or they necessitate further unstated assumptions such as backwards causality.
First, let us assume (a) but not (b), which can be formulated as follows: Omega, a computer engineer, can read your code and test run it as many times as he would like in advance. You must submit (simple, unobfuscated) code which either chooses to one- or two-box. The contents of the boxes will depend on Omega's prediction of your code's choice. Do you submit one- or two-boxing code?
Second, let us assume (b) but not (a), which can be formulated as follows: Omega has subjected you to the Newcomb's setup, but because of a bug in its code, its prediction is based on someone else's choice than yours, which has no correlation with your choice whatsoever. Do you one- or two-box?
Both of these formulations translate straightforwardly into payoff matrices and any sort of sensible decision theory you throw at them give the correct solution. The paradox disappears when the ambiguity between the two above possibilities are removed. As far as I can see, all disagreement between one-boxers and two-boxers are simply a matter of one-boxers choosing the first and two-boxers choosing the second interpretation. If so, Newcomb's paradox is not as much interesting as poorly specified. The supposed superiority of TDT over CDT either relies on the paradox not reducing to either of the above or by fiat forcing CDT to work with the wrong payoff matrices.
I would be interested to see an unambiguous and nontrivial formulation of the paradox.
Some quick and messy addenda:
Thanks for this post; it articulates many of the thoughts I've had on the apparent inconsistency of common decision-theoretic paradoxes such as Newcomb's problem. I'm not an expert in decision theory, but I have a computer science background and significant exposure to these topics, so let me give it a shot.
The strategy I have been considering in my attempt to prove a paradox inconsistent is to prove a contradiction using the problem formulation. In Newcomb's problem, suppose each player uses a fair coin flip to decide whether to one-box or two-box. Then Omega could not have a sustained correct prediction rate above 50%. But the problem formulation says Omega does; therefore the problem must be inconsistent.
Alternatively, Omega knew the outcome of the coin flip in advance; let's say Omega has access to all relevant information, including any supposed randomness used by the decision-maker. Then we can consider the decision to already have been made; the idea of a choice occurring after Omega has left is illusory (i.e. deterministic; anyone with enough information could have predicted it.) Admittedly, as you say quite eloquently:
In this case of the all-knowing Omega, talking about what someone should choose after Omega has left seems mistaken. The agent is no longer free to make an arbitrary decision at run-time, since that would have backwards causal implications; we can, without restricting which algorithm is chosen, require the decision-making algorithm to be written down and provided to Omega prior to the whole simulation. Since Omega can predict the agent's decision, the agent's decision does determine what's in the box, despite the usual claim of no causality. Taking that into account, CDT doesn't fail after all.
It really does seem to me like most of these supposed paradoxes of decision theory have these inconsistent setups. I see that wedrifid says of coin flips:
I would love to hear from someone in further detail on these issues of consistency. Have they been addressed elsewhere? If so, where?
This seems like a worthy approach to paradoxes! I'm going to suggest the possibility of broadening your search slightly. Specifically, to include the claim "and this is paradoxical" as one of the things that can be rejected as producing contradictions. Because in this case there just isn't a paradox. You take the one box, get rich and if there is a decision theory that says to take both boxes you get a better theory. For this reason "Newcomb's Paradox" is a misnomer and I would only use "Newcomb's Problem" as an acceptable name.
Yes, if the player is allowed access to entropy that Omega cannot have then it would be absurd to also declare that Omega can predict perfectly. If the coin flip is replaced with a quantum coinflip then the problem becomes even worse because it leaves an Omega that can perfectly predict what will happen but is faced with a plainly inconsistent task of making contradictory things happen. The problem specification needs to include a clause for how 'randomization' is handled.
Here is where I should be able to link you to the wiki page on free will where you would be given an explanation of why the notion that determinism is incompatible with choice is a confusion. Alas that page still has pretentious "Find Out For Yourself" tripe on it instead of useful content. The wikipedia page on compatibilism is somewhat useful but not particularly tailored to a reductionist decision theory focus.
There have been attempts to create derivatives of CDT that work like that. That replace the "C" from conventional CDT with a type of causality that runs about in time as you mention. Such decision theories do seem to handle most of the problems that CDT fails at. Unfortunately I cannot recall the reference.
I'm not sure which further details you are after. Are you after a description of Newcomb's problem that includes the details necessary to make it consistent? Or about other potential inconsistencies? Or other debates about whether the problems are inconsistent?
Thanks for the response! I'm looking for a formal version of the viewpoint you reiterated at the beginning of your most recent comment:
That makes a lot of sense, but I haven't been able to find it stated formally. Wolpert and Benford's papers (using game theory decision trees or alternatively plain probability theory) seem to formally show that the problem formulation is ambiguous, but they are recent papers, and I haven't been able to tell how well they stand up to outside analysis.
If there is a consensus that the sufficient use of randomness prevents Omega from having perfect or nearly perfect predictions, then why is Newcomb's problem still relevant? If there's no randomness, wouldn't an appropriate application of CDT result in one-boxing since the decision-maker's choice and Omega's prediction are both causally determined by the decision-maker's algorithm, which was fixed prior to the making of the decision?
I'm curious: why can't normal CDT handle it by itself? Consider two variants of Newcomb's problem:
(Note: I'm out of my depth here, and I haven't given a great deal of thought to precommitment and the possibility of allowing algorithms to rewrite themselves.)
You can consider an ideal agent that uses argmax E to find what it chooses, where E is some environment function . Then what you arrive at is that argmax gets defined recursively - E contains argmax as well - and it just so happens that the resulting expression is only well defined if there's nothing in the first box and you choose both boxes. I'm writing a short paper about that.
You may be thinking of Huw Price's paper available here
I agree; wherever there is paradox and endless debate, I have always found ambiguity in the initial posing of the question. An unorthodox mathematician named Norman Wildberger just released a new solution by unambiguously specifying what we know about Omega's predictive powers.
I seems to me that what he gives is not so much a new solution as a neat generalized formulation. His formula gives you different results depending on whether you're a causal decision theorist or not.
The causal decision theorist will say that his pA should be considered to be P(prediction = A|do(A)) and pB is P(prediction = B|do(B)), which will, unless you assume backward causation, just be P(prediction = A) and P(prediction = B) and thus sum to 1, hence the inequality at the end doesn't hold and you should two-box.
I do not agree that a CDT must conclude that P(A)+P(B) = 1. The argument only holds if you assume the agent's decision is perfectly unpredictable, i.e. that there can be no correlation between the prediction and the decision. This contradicts one of the premises of Newcomb's Paradox, which assumes an entity with exactly the power to predict the agent's choice. Incidentally, this reduces to the (b) but not (a) from above.
By adopting my (a) but not (b) from above, i.e. Omega as a programmer and the agent as predictable code, you can easily see that P(A)+P(B) = 2, which means one-boxing code will perform the best.
Further elaboration of the above:
Imagine John, who never understood how the days of the week succeed each other. Rather, each morning, a cab arrives to take him to work if it is a work day, else he just stays at home. Omega must predict if he will go to work or not the before the cab would normally arrive. Omega knows that weekdays are generally workdays, while weekends are not, but Omega does not know the ins and outs of particular holidays such as fourth of July. Omega and John play this game each day of the week for a year.
Tallying the results, John finds that the score is as follows: P( Omega is right | I go to work) = 1.00, P( Omega is right | I do not go to work) = 0.85, which sums to 1.85. John, seeing that the sum is larger than 1.00, concludes that Omega seems to have rather good predictive power about whether he will go to work, but is somewhat short of perfect accuracy. He realizes that this has a certain significance for what bets he should take with Omega, regarding whether he will go to work tomorrow or not.
But that's not CDT reasoning. CDT uses surgery instead of conditionalization, that's the whole point. So it doesn't look at P(prediction = A|A), but at P(prediction = A|do(A)) = P(prediction = A).
Your example with the cab doesn't really involve a choice at all, because John's going to work is effectively determined completely by the arrival of the cab.
I am not sure where our disagreement lies at the moment.
Are you using choice to signify strongly free will? Because that means the hypothetical Omega is impossible without backwards causation, leaving us at (b) but not (a) and the whole of Newcomb's paradox moot. Whereas, if you include in Newcomb's paradox, the choice of two-boxing will actually cause the big box to be empty, whereas the choice of one-boxing will actually cause the big box to contain a million dollars by a mechanism of backwards causation, then any CDT model will solve the problem.
Perhaps we can narrow down our disagreement by taking the following variation of my example, where there is at least a bit more of choice involved:
Imagine John, who never understood why he gets thirsty. Despite there being a regularity in when he chooses to drink, this is for him a mystery. Every hour, Omega must predict whether John will choose to drink within the next hour. Omega's prediction is made secret to John until after the time interval has passed. Omega and John play this game every hour for a month, and it turns out that while far from perfect, Omega's predictions are a bit better than random. Afterwards, Omega explains that it beats blind guesses by knowing that John will very rarely wake up in the middle of the night to drink, and that his daily water consumption follows a normal distribution with a mean and standard deviation that Omega has estimated.
I'm not entirely sure either. I was just saying that a causal decision theorist will not be moved by Wildberger's reasoning, because he'll say that Wildberger is plugging in the wrong probabilities: when calculating an expectation, CDT uses not conditional probability distributions but surgically altered probability distributions. You can make that result in one-boxing if you assume backwards causation.
I think the point we're actually talking about (or around) might be the question of how CDT reasoning relates to you (a). I'm not sure that the causal decision theorist has to grant that he is in fact interpreting the problem as "not (a) but (b)". The problem specification only contains the information that so far, Omega has always made correct predictions. But the causal decision theorist is now in a position to spoil Omega's record, if you will. Omega has already made a prediction, and whatever the causal decision theorist does now isn't going to change that prediction. The fact that Omega's predictions have been absolutely correct so far doesn't enter into the picture. It just means that for all agents x that are not the causal decision theorist, P(x does A|Omega predicts that x does A) = 1 (and the same for B, and whatever value than 1 you might want for an imperfect predictor Omega).
About the way you intend (a), the causal decision theorist would probably say that's backward causation and refuse to accept it.
One way of putting it might be that the causal decision theorist simply has no way of reasoning with the information that his choice is predetermined, which is what I think you intend to convey with (a). Therefore, he has no way of (hypothetically) inferring Omega's prediction from his own (hypothetical) action (because he's only allowed to do surgery, not conditionalization).
No, actually. Just the occurrence of a deliberation process whose outcome is not immediately obvious. In both your examples, that doesn't happen: John's behavior simply depends on the arrival of the cab or his feeling of thirst, respectively. He doesn't, in a substantial sense, make a decision.
(Thanks for discussing!)
I will address your last paragraph first. The only significant difference between my original example and the proper Newcomb's paradox is that, in Newcomb's paradox, Omega is made a predictor by fiat and without explanation. This allows perfect prediction and choice to sneak into the same paragraph without obvious contradiction. It seems, if I try to make the mode of prediction transparent, you protest there is no choice being made.
From Omega's point of view, its Newcomb subjects are not making choices in any substantial sense, they are just predictably acting out their own personality. That is what allows Omega its predictive power. Choice is not something inherent to a system, but a feature of an outsider's model of a system, in much the same sense as random is not something inherent to a Eeny, meeny, miny, moe however much it might seem that way to children.
As for the rest of our disagreement, I am not sure why you insist that CDT must work with a misleading model. The standard formulation of Newcomb's paradox is inconsistent or underspecified. Here are some messy explanations for why, in list form:
Not if you're a compatibilist, which Eliezer is last I checked.
The post scav made more or less represents my opinion here. Compatibilism, choice, free will and determinism are too many vague definitions for me to discuss with. For compatibilism to make any sort of sense to me, I would need a new definition of free will. It is already difficult to discuss how stuff is, without simultaneously having to discuss how to use and interpret words.
Trying to leave the problematic words out of this, my claim is that the only reason CDT ever gives a wrong answer in a Newcomb's problem is that you are feeding it the wrong model. http://lesswrong.com/lw/gu1/decision_theory_faq/8kef elaborates on this without muddying the waters too much with the vaguely defined terms.
I don't think compatibilist means that you can pretend two logically mutually exclusive propositions can both be true. If it is accepted as a true proposition that Omega has predicted your actions, then your actions are decided before you experience the illusion of "choosing" them. Actually, whether or not there is an Omega predicting your actions, this may still be true.
Accepting the predictive power of Omega, it logically follows that when you one-box you will get the $1M. A CDT-rational agent only fails on this if it fails to accept the prediction and constructs a (false) causal model that includes the incoherent idea of "choosing" something other than what must happen according to the laws of physics. Does CDT require such a false model to be constructed? I dunno. I'm no expert.
The real causal model is that some set of circumstances decided what you were going to "choose" when presented with Omega's deal, and those circumstances also led to Omega's 100% accurate prediction.
If being a compatibilist leads you to reject the possibility of such a scenario, then it also logically excludes the perfect predictive power of Omega and Newcomb's problem disappears.
But in the problem as stated, you will only two-box if you get confused about the situation or you don't want $1M for some reason.
Regarding illegal choices, the transparent variation makes it particularly clear, i.e. you can't take both boxes if you see a million in first box, and take 1 box otherwise.
You can walk backwards from your decision to the point where a copy of you had been made, and then forward to the point where a copy is processed by the Omega, to find the relation of your decision to the box state causally.
I agree with the content, though I am not sure if I approve of a terminology where causation traverses time like a two-way street.
I probably wasn't expressing myself quite clearly. I think the difference is this: Newcomb subjects are making a choice from their own point of view. Your Johns aren't really make a choice even from their internal perspective: they just see if the cab arrives/if they're thirsty and then without deliberation follow what their policy for such cases prescribes. I think this difference is substantial enough intuitively so that the John cases can't be used as intuition pumps for anything relating to Newcomb's.
I don't think it is, actually. It just seems so because it presupposes that your own choice is predetermined, which is kind of hard to reason with when you're right in the process of making the choice. But that's a problem with your reasoning, not with the scenario. In particular, the CDT agent has a problem with conceiving of his own choice as predetermined, and therefore has trouble formulating Newcomb's problem in a way that he can use - he has to choose between getting two-boxing as the solution or assuming backward causation, neither of which is attractive.
Then I guess I will try to leave it to you to come up with a satisfactory example. The challenge is to include Newcomblike predictive power for Omega, but not without substantiating how Omega achieves this, while still passing your own standards of subject makes choice from own point of view. It is very easy to accidentally create paradoxes in mathematics, by assuming mutually exclusive properties for an object, and the best way to discover these is generally to see if it is possible construct or find an instance of the object described.
This is not a failure of CDT, but one of your imagination. Here is a simple, five minute model which has no problems conceiving Newcomb's problem without any backwards causation:
You can pretend to enter this situation at T=4 as suggested by the standard Newcomb's problem. Then you can use the dominance principle and you will lose. But this just using a terrible model. You entered at T=0, because you were needed at T=1 for Omega's inspection. If you did not enter the situation at T=0, then you can freely make a choice C at T=5 without any correlation to P, but that is not Newcomb's problem.
Instead, at T=4 you become aware of the situation, and your decision making algorithm must return a value for C. If you consider this only from T=4 and onward, this is completely uninteresting, because C is already determined. At T=1, P was determined to either P1 or P2, and the value of C follow directly from this. Obviously, healthy one-boxing code wins and unhealthy two-boxing code loses, but there is no choice being made here, just different code with different return values being rewarded differently, and that is not Newcomb's problem either.
Finally, we will work under illusion of choice with Omega as a perfect predictor. We realize that T=0 is the critical moment, seeing as all subsequent T follows directly from this. We work backwards as follows:
Shorthand version of all the above; if the decision is necessarily predetermined before T=4, then you should not pretend you make the decision at T=4. Insert a decision making step at T=0.5, which causally determines the value of P and C. Apply your CDT to this step.
This is the only way of doing CDT honestly, and it is the slightest bit messy, but that is exactly what happens when you create a reference to the decision the decision theory is going to make in the future in the problem itself with perfect correlation to the decision before the decision has overtly been made. This sort of self reference creates impossibilities out of the thin air every day of week, such as when Pinocchio says my nose will grow now. The good news is that this way of doing it is a lot less messy than inventing a new, superfluous decision theory, and it also allows you to deal with problems like the psychopath button without any trouble whatsoever.
I'm with incogn on this one: either there is predictability or there is choice; one cannot have both.
Incogn is right in saying that, from omega's point of view, the agent is purely deterministic, i.e. more or less equivalent to a computer program. Incogn is slightly off-the-mark in conflating determinism with predictability: a system can be deterministic, but still not predictable; this is the foundation of cryptography. Deterministic systems are either predictable or are not. Unless Newcombs problem explicitly allows the agent to be non-deterministic, but this is unclear.
The only way a deterministic system becomes unpredictable is if it incorporates a source of randomness that is stronger than the ability of a given intelligence to predict. There are good reasons to believe that there exist rather simple sources of entropy that are beyond the predictive power of any fixed super-intelligence -- this is not just the foundation of cryptography, but is generically studied under the rubric of 'chaotic dynamical systems'. I suppose you also have to believe that P is not NP. Or maybe I should just mutter 'Turing Halting Problem'. (unless omega is taken to be a mythical comp-sci "oracle", in which case you've pushed decision theory into that branch of set theory that deals with cardinal numbers larger than the continuum, and I'm pretty sure you are not ready for the dragons that lie there.)
If the agent incorporates such a source of non-determinism, then omega is unable to predict, and the whole paradox falls down. Either omega can predict, in which case EDT, else omega cannot predict, in which case CDT. Duhhh. I'm sort of flabbergasted, because these points seem obvious to me ... the Newcomb paradox, as given, seems poorly stated.
Think of real people making choices and you'll see it's the other way around. The carefully chosen paths are the predictable ones if you know the variables involved in the choice. To be unpredictable, you need think and choose less.
Hell, the archetypical imagery of someone giving up on choice is them flipping a coin or throwing a dart with closed eyes -- in short resorting to unpredictability in order to NOT choose by themselves.
Either your claim is false or you are using a definition of at least one of those two words that means something different to the standard usage.
Newcomb's problem makes the stronger precondition that the agent is both predictable and that in fact one action has been predicted. In that specific situation, it would be hard to argue against that one action being determined and immutable, even if in general there is debate about the relationship between determinism and predictability.
I think I agree, by and large, despite the length of this post.
Whether choice and predictability are mutually exclusive depends on what choice is supposed to mean. The word is not exactly well defined in this context. In some sense, if variable > threshold then A, else B is a choice.
I am not sure where you think I am conflating. As far as I can see, perfect prediction is obviously impossible unless the system in question is deterministic. On the other hand, determinism does not guarantee that perfect prediction is practical or feasible. The computational complexity might be arbitrarily large, even if you have complete knowledge of an algorithm and its input. I can not really see the relevance to my above post.
Finally, I am myself confused as to why you want two different decision theories (CDT and EDT) instead of two different models for the two different problems conflated into the single identifier Newcomb's paradox. If you assume a perfect predictor, and thus full correlation between prediction and choice, then you have to make sure your model actually reflects that.
Let's start out with a simple matrix, P/C/1/2 are shorthands for prediction, choice, one-box, two-box.
If the value of P is unknown, but independent of C: Dominance principle, C=2, entirely straightforward CDT.
If, however, the value of P is completely correlated with C, then the matrix above is misleading, P and C can not be different and are really only a single variable, which should be wrapped in a single identifier. The matrix you are actually applying CDT to is the following one:
The best choice is (P&C)=1, again by straightforward CDT.
The only failure of CDT is that it gives different, correct solutions to different, problems with a properly defined correlation of prediction and choice. The only advantage of EDT is that it is easier to cheat in this information without noticing it - even when it would be incorrect to do so. It is entirely possible to have a situation where prediction and choice are correlated, but the decision theory is not allowed to know this and must assume that they are uncorrelated. The decision theory should give the wrong answer in this case.
If Omega cannot predict, TDT will two-box.
Thanks for the link.
I like how he just brute forces the problem with (simple) mathematics, but I am not sure if it is a good thing to deal with a paradox without properly investigating why it seems to be a paradox in the first place. It is sort of like saying that this super convincing card trick you have seen, there is actually no real magic involved without taking time to address what seems to require magic and how it is done mundanely.
I think this is a very clear account of the issues with these problems.
I like your explanations of how correct model choice leads to CDT getting it right all the time; similarly it seems correct model choice should let EDT get it right all the time. In this light CDT and EDT are really heuristics for how to make decisions with simplified models.