= 27d567bc2d48d9f40e9ccc20ea370b15)
taw comments on Counterfactual Mugging - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (257)
I really fail to see why you're all so fascinated by Newcomb-like problems. When you break causality, all logic based on causality doesn't function any more. If you try to model it mathematically, you will get inconsistent model always.
They don't require breaking causality. The argument works if Omega is barely predicting you above chance. I'm sure there are plenty of normal people who can do that just by talking to you.
There are also more important reasons. Take the doomsday argument. You can use the fact that you're alive now to predict that we'll die out "soon". Suppose you had a choice between saving a life in a third-world country that likely wouldn't amount to anything, or donating to SIAI to help in the distant future. You know it's very unlikely for there to be a distant future. It's like Omega did his coin toss, and if it comes up tails, we die out early and he asks you to waste the money by donating to SIAI. If it comes up heads, you're in the future, and it's better if you would have donated.
That's not some thing that might happen. That's a decision you have to make before you pick a charity to donate to. Lives are riding on this. That's if the coin lands on tails. If it lands on heads, there is more life riding on it than has so far existed in the known universe. Please choose carefully.
Arguments like these remind me of students' mistakes from Algorithms and Data Structures 101 - statements like that are very intuitive, absolutely wrong, and once you figure out why this reasoning doesn't work it's easy to forget that most people didn't go through this ever.
What is required is Omega predicting better than chance in the worst case. Predicting correctly with ridiculously tiny chance of error against "average" person is worthless.
To avoid Omega and causality silliness, and just demonstrate this intuition - let's take a slightly modified version of Boolean satisfiability - but instead of one formula we have three formulas of the same length. If all three are identical, return true or false depending on its satisfiability, if they're different return true if number of one bits in problem is odd (or some other trivial property).
It is obviously NP-complete, as any satisfiability problem reduces to it by concatenating it three times. If we use exponential brute force to solve the hard case, average running time is O(n) for scanning the string plus O(2^(n/3)) for brute forcing but only 2^-(2n/3) of the time, that is O(1). So we can solve NP-complete problems in average linear time.
What happened? We were led astray by intuition, and assumed that problems that are difficult in worst case cannot be trivial on average. But this equal weighting is an artifact - if you tried reducing any other NP problem into this, you'd be getting very difficult ones nearly all the time, as if by magic.
Back to Omega - even if Omega predicts normal people very well, as long as there are any thinking being who is cannot predict - Omega must break causality. And such being are not just hypothetical - people who decide based on a coin toss are exactly like that. Silly rules about disallowing chance merely make counterexamples more complicated, Omega and Newcomb are still as much based on sloppy thinking as ever.
I don't know any reason why a coin toss would be the best choice in Newcomb's paradox. If you decide based on reason, and don't decide to flip a coin, and Omega knows you well, he can predict your action above chance. The paradox stands.
There's no need to break causality. You are a being implemented in chaotic wetware. However, there's no reason to think we couldn't have rational agents implemented in much more predictable form, as python routines for example, so that any being with superior computation power could simply inspect the source and determine what the output would be.
In such a case, Newcomb-like problems would arise, perfectly lawfully, under normal physics.
In fact, Newcomb-like problems fall naturally out of any ability to simulate and predict the actions of other agents. Omega as described is essentially the limit as predictive power goes to infinity.
If we define an imperfect predictor as a perfect predictor plus noise, i.e. produces the correct prediction with probability p regardless of the cognition algorithm it's trying to predict, then Newcomb-like problems are very robust to imperfect prediction: for any p > .5 there is some payoff ratio great enough to preserve the paradox, and the required ratio goes down as the prediction improves. e.g. if 1-boxing gets 100 utilons and 2-boxing gets 1 utilon, then the predictor only needs to be more than 50.5% accurate. So the limit in that direction favors 1-boxing.
What other direction could there be? If the prediction accuracy depends on the algorithm-to-be-predicted (as it would in the real world), then you could try to be an algorithm that is mispredicted in your favor... but a misprediction in your favor can only occur if you actually 2-box, so it only takes a modicum of accuracy before a 1-boxer who tries to be predictable is better off than a 2-boxer who tries to be unpredictable.
I can't see any other way for the limit to turn out.
If you have two agents trying to precommit not to be blackmailed by each other / precommit not to pay attention to the others precommitment, then any attempt to take a limit of this Newcomblike problem does depend on how you approach the limit. (I don't know how to solve this problem.)
The value(s) for which the limit is being taken here is unidirectional predictive power, which is loosely a function of the difference in intelligence between the two agents; intuitively, I think a case could be made that (assuming ideal rationality) the total accuracy of mutual behavior prediction between two agents is conserved in some fashion, that doubling the predictive power of one unavoidably would roughly halve the predictive power of the other. Omega represents an entity with a delta-g so large vs. us that predictive power is essentially completely one-sided.
From that basis, allowing the unidirectional predictive power of both agents to go to infinity is probably inherently ill-defined and there's no reason to expect the problem to have a solution.
You cannot do that without breaking Rice's theorem. If you assume you can find out the answer from someone else's source code -> instant contradiction.
You cannot work around Rice's theorem or around causality by specifying 50.5% accuracy independently of modeled system, any accuracy higher than 50%+epsilon is equivalent to indefinitely good accuracy by repeatedly predicting (standard cryptographic result), and 50%+epsilon doesn't cause the paradox.
Give me one serious math model of Newcomb-like problems where the paradox emerges while preserving causality. Here are some examples. Then you model it, you either get trivial solution to one-box, or causality break, or omega loses. * You decide first what you would do in every situation, omega decides second, and now you only implement your initial decision table and are not allowed to switch. Game theory says you should implement one-boxing. * You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. Game theory says you should precommit to one-box, then implement two-boxing, omega loses. * You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. If omega always decides correctly, then he bases his decision on your switch, which either turns it into model #1 (you cannot really switch, precommitment is binding), or breaks causality.
Rice's theorem says you can't predict every possible algorithm in general. Plenty of particular algorithms can be predictable. If you're running on a classical computer and Omega has a copy of you, you are perfectly predictable.
And all of your choices are just as real as they ever were, see the OB sequence on free will (I think someone referred to it already).
And the argument that omega just needs predictive power of 50.5% to cause the paradox only works if it works against ANY arbitrary algorithm. Having that power against any arbitrary algorithm breaks Rice's Theorem, having that power (or even 100%) against just limited subset of algorithms doesn't cause the paradox.
If you take strict decision tree precommitment interpretation, then you fix causality. You decide first, omega decides second, game theory says one-box, problem solved.
Decision tree precommitment is never a problem in game theory, as precommitment of the entire tree commutes with decisions by other agents:
are identical, as B cannot decide based on f. So the changing your mind problem never occurs.
With omega:
I don't think the paradox exist in any plausible mathematization of the problem. It looks to me like another of those philosophical problems that exist because of sloppiness of natural language and very little more, I'm just surprised that OB/LW crowd cares about this one and not about others. OK, I admit I really enjoyed it the first time I saw it but just as something fun, nothing more than that.
I don't know why nobody mentioned this at the time, but that's hardly an unpopular view around here (as I'm sure you've noticed by now).
The interesting thing about Newcomb had nothing to do with thinking it was a genuine paradox - just counterintuitive for some.
Whaddaya mean humans are rational agents with access to genuine randomness? That's what we're arguing about in the first place!
Perhaps Omega is entangled with your brain such that in all the worlds in which you would choose to one-box, he would predict that you one-box, and all the worlds in which you would choose to two-box, he would predict that you two-box?
In the original formulation, if Omega expects you to flip a coin, he leaves box B empty.
You wouldn't know this with certainty* because it wouldn't be true.
(*unless you were delusional)
The guy next door is on roughly your mental level. Thus, the guy next door can't predict your actions perfectly, because he can't run a perfect simulation of your mind that's faster than you. He doesn't have the capacity.
And he certainly doesn't have the capacity to simulate the environment, including other people, while doing so.
Humans may or may not generally have access to genuine randomness.
It's as yet unknown whether we even have run on quantum randomness; and its also unprovable that quantum randomness is actually genuine randomness, and not just based on effects we don't yet understand, as so many other types of randomness have been.
You're not taking this in the least convenient possible world. Surely it's not impossible in principle that your neighbor can simulate you and your environment. Perhaps your neighbor is superintelligent?
It's ALSO not impossible in principle in the real world. A superintelligent entity could, in principle, perfectly predict my actions. Remember, in the Least Convenient Possible World quantum "randomness" isn't random.
As such, this ISN'T a fundamental difference between humans and "such beings". Which was all I set out to demonstrate.
I was using the "most plausible world" on the basis that it seemed pretty clear that that was the one Roko intended. (Where your neighbour isn't in fact Yahweh in disguise). EDIT: Probably should specify worlds for things in this kind of environment. Thanks, the critical environment here is helping me think about how I think/argue.
On second thoughts, since many clever philosophers spend careers on these problems, I may be missing something.
Nah, they just need something to talk about.
I find the problem interesting, so I'll try to explain why I find it interesting.
So there are these blogs called Overcoming Bias and Less Wrong, and the people posting on it seem like very smart people, and they say very reasonable things. They offer to teach how to become rational, in the sense of "winning more often". I want to win more often too, so I read the blogs.
Now a lot of what these people are saying sounds very reasonable, but it's also clear that the people saying these things are much smarter than me; so much so that although their conclusions sound very reasonable, I can't always follow all the arguments or steps used to reach those conclusions. As part of my rationalist training, I try to notice when I can follow the steps to a conclusion, and when I can't, and remember which conclusions I believe in because I fully understand it, and which conclusions I am "tentatively believing in" because someone smart said it, and I'm just taking their word for it for now.
So now Vladimir Nesov presents this puzzle, and I realize that I must not have understood one of the conclusions (or I did understand them, and the smart people were mistaken), because it sounds like if I were to follow the advice of this blog, I'd be doing something really stupid (depending on how you answered VN's problem, the stupid thing is either "wasting $100" or "wasting $4950").
So how do I reconcile this with everything I've learned on this blog?
Think of most of the blog as a textbook, with VN's post being an "exercise to the reader" or a "homework problem".
The primary reason for resolving Newcomb-like problems is to explore the fundamental limitations of decision theories.
It sounds like you are still confused about free will. See Righting a Wrong Question, Possibility and Could-ness, and Daniel Dennett's lecture here.
I think I'm not confused about free will, and that the links I gave should help to resolve most of the confusion. Maybe you should write a blog post/LW article where you formulate the nature of your confusion (if you still have it after reading the relevant material), I'll respond to that.
Not really - all that is neccessary is that Omega is a sufficiently accurate predictor that the payoff matrix, taking this accuracy into question, still amounts to a win for the given choice. There is no need to be a perfect predictor. And if an imperfect, 99.999% predictor violates free will, then it's clearly a lost cause anyway (I can predict with similar precision many behaviours about people based on no more evidence than their behaviour and speech, never mind godlike brain introspection) Do you have no "choice" in deciding to come to work tomorrow, if I predict based on your record that you're 99.99% reliable? Where is the cut-off that free will gets lost?
Chances are I can predict such a response too, and so won't tell you of my prediction (or tell you in such a way that you will be more likely to attend: eg. "I've a $50 bet you'll attend tomorrow. Be there and I'll split it 50:50"). It doesn't change the fact that in this particular instance I can fortell the future with a high degree of accuracy. Why then would it violate free will if Omega could predict your accuracy in this different situation (one where he's also able to predict the effects of him telling you) to a similar precision?
Then take my bet situation. I announce your attendance, and cut you in with a $25 stake in attendance. I don't think it would be unusual to find someone who would indeed appear 99.99% of the time - does that mean that person has no free will?
People are highly, though not perfectly, predictable under a large number of situations. Revealing knowledge about the prediction complicates things by adding feedback to the system, but there are lots of cases where it still doesn't change matters much (or even increases predictability). There are obviously some situations where this doesn't happen, but for Newcombe's paradox, all that is needed is a predictor for the particular situation described, not any general situation. (In fact Newcombe's paradox is equally broken by a similar revelation of knowledge. If Omega were to reveal its prediction before the boxes are chosen, a person determined to do the opposite of that prediction opens it up to a simple Epimenides paradox.)