Counterfactual Mugging
Related to: Can Counterfactuals Be True?, Newcomb's Problem and Regret of Rationality.
Imagine that one day, Omega comes to you and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But see, the Omega tells you that if the coin came up heads instead of tails, it'd give you $10000, but only if you'd agree to give it $100 if the coin came up tails.
Omega can predict your decision in case it asked you to give it $100, even if that hasn't actually happened, it can compute the counterfactual truth. The Omega is also known to be absolutely honest and trustworthy, no word-twisting, so the facts are really as it says, it really tossed a coin and really would've given you $10000.
From your current position, it seems absurd to give up your $100. Nothing good happens if you do that, the coin has already landed tails up, you'll never see the counterfactual $10000. But look at this situation from your point of view before Omega tossed the coin. There, you have two possible branches ahead of you, of equal probability. On one branch, you are asked to part with $100, and on the other branch, you are conditionally given $10000. If you decide to keep $100, the expected gain from this decision is $0: there is no exchange of money, you don't give Omega anything on the first branch, and as a result Omega doesn't give you anything on the second branch. If you decide to give $100 on the first branch, then Omega gives you $10000 on the second branch, so the expected gain from this decision is
-$100 * 0.5 + $10000 * 0.5 = $4950
So, this straightforward calculation tells that you ought to give up your $100. It looks like a good idea before the coin toss, but it starts to look like a bad idea after the coin came up tails. Had you known about the deal in advance, one possible course of action would be to set up a precommitment. You contract a third party, agreeing that you'll lose $1000 if you don't give $100 to Omega, in case it asks for that. In this case, you leave yourself no other choice.
But in this game, explicit precommitment is not an option: you didn't know about the Omega's little game until the coin was already tossed and the outcome of the toss was given to you. The only thing that stands between Omega and your 100$ is your ritual of cognition. And so I ask you all: is the decision to give up $100 when you have no real benefit from it, only counterfactual benefit, an example of winning?
P.S. Let's assume that the coin is deterministic, that in the overwhelming measure of the MWI worlds it gives the same outcome. You don't care about a fraction that sees a different result, in all reality the result is that Omega won't even consider giving you $10000, it only asks for your $100. Also, the deal is unique, you won't see Omega ever again.
Loading…
Comments (144)
The counterfactual anti-mugging: One day No-mega appears. No-mega is completely trustworthy etc. No-mega describes the counterfactual mugging to you, and predicts what you would have done in that situation not having met No-mega, if Omega had asked you for $100.
If you would have given Omega the $100, No-mega gives you nothing. If you would not have given Omega $100, No-mega gives you $10000. No-mega doesn't ask you any questions or offer you any choices. Do you get the money? Would an ideal rationalist get the money?
Okay, next scenario: you have a magic box with a number p inscribed on it. When you open it, either No-mega comes out (probability p) and performs a counterfactual anti-mugging, or Omega comes out (probability 1-p), flips a fair coin and proceeds to either ask for $100, give you $10000, or give you nothing, as in the counterfactual mugging.
Before you open the box, you have a chance to precommit. What do you do?
I would have no actionable suspicion that I should give Omega the $100 unless I knew about No-mega. So I get the $10000 only if No-mega asks the question "What would Eliezer do knowing about No-mega?" and not if No-mega asks the question "What would Eliezer do not knowing about No-mega?"
Do you have a point?
Yes, that there can just as easily be a superintelligence that rewards people predicted to act one way as one that rewards people predicted to act the other. Which precommitment is most rational depends depends on the which type you expect to encounter.
I don't expect to encounter either, and on the other hand I can't rule out fallible human analogues of either. So for now I'm not precommitting either way.
Yeah, now. But after Omega really, really, appears in front of you, chance of Omega existing is about 1. Chance of No-Mega is still almost non-existent. In this problem, existence of Omega is given. It's not something you are expecting to encounter now, just as we're not expecting to encounter eccentric Kavkan billionaires that will give you money for toxicating yourself. The Kavka's Toxin and the counterfactual mugging present a scenario that is given, and ask you how would you act then.
You don't precommit to "give away the $100, to anyone who asks". You precommit to give away the $100 in exactly the situation I described. Or, generalizing such precommitments, you just compute your decisions on the spot, in a reflectively consistent fashion. If that's what you want do to with your future self, that is.
This problem seems conceptually identical to Kavka's toxin puzzle; we have merely replaced intending to drink the poison/pay $100 with being the sort of person whom Omega would predict would do it.
Since, as has been pointed out, one needn't be a perfect predictor for the game to work, I think I'll actually try this on some of my friends.
Thanks for reminding of the Kavka's puzzle. I think that puzzle is unnecessarily mental in its formulation, for example you have to "intend". It's less confusing when you work on more technical concepts of decision-making, evidence, preference and precommitment.
I can't imagine how you are going to perform this on your friends...
The main problem, I think, is getting them to believe that I'm a reliable predictor (i.e. that I predict as well as I claim I do).
Actually, I don't know that if I do this it will show anything relevant to the problem under consideration. But I think it will show something. It has in fact already shown that I believe that 59% of them would agree to give me the money, either because they are sufficiently similar to Eliezer, or because they enjoy random acts of silliness (and the amount of money involved will be pretty trivial).
I don't see the difficulty. No, you don't win by giving Omega $100. Yes, it would have been a winning bet before the flip if, as you specify, the coin is fair. Your PS, in which you say to "assume that in the overwhelming measure of the MWI worlds it gives the same outcome", contradicts the assertion that the coin is fair, and so you have asked us for an answer to an incoherent question.
The difficulty comes from projecting the ideal decision theory on people. Look how many people are ready to pay up $100, so it must be a real difficulty.
The fairness of a coin is a property of your mind, not of the coin itself. The coin can be fair in a deterministic world, the same way you can have free will in deterministic world.
Better to say that your state of knowledge about the coin, prior to Omega appearing, is that it has a probability 1/2 of being heads and 1/2 of being tails. The MWI clause is supposed to make the problem harder by preventing you from assigning utility (once Omega appears) to your 'other selves' in other Everett branches. The problem is then just: "how, knowing that Omega might appear, but not knowing what the coin flip will be, can I maximise my utility?" If Omega appears in front of you right now then that's a different question.
I convinced myself to one-box in Newcomb by simply treating it as if the contents of the boxes magically change when I made my decision. Simply draw the decision tree and maximize u-value.
I convinced myself to cooperate in the Prisoner's Dilemma by treating it as if whatever decision I made the other person would magically make too. Simply draw the decision tree and maximize u-value.
It seems that Omega is different because I actually have the information, where in the others I don't.
For example, In Newcomb, if we could see the contents of both boxes, then I should two-box, no? In the Prisoner's Dilemma, if my opponent decides before me and I observe the decision, then I should defect, no?
I suspect that this means that my thought process in Newcomb and the Prisoner's Dilemma is incorrect. That there is a better way to think about them that makes them more like Omega. Am I correct? Does this make sense?
Yes, the objective in designing this puzzle was to construct an example where according to my understanding of the correct way to make decision, the correct decision looks like losing. In other cases you may say that you close your eyes, pretend that your decision determines the past or other agents' actions, and just make the decision that gives the best outcome. In this case, you choose the worst outcome. The argument is that on reflection it still looks like the best outcome, and you are given an opportunity to think about what's the correct perspective from which it's the best outcome. It binds the state of reality to your subjective perspective, where in many other thought experiments you may dispense with this connection and focus solely on the reality, without paying any special attention to the decision-maker.
In Newcomb, before knowing the box contents, you should one-box. If you know the contents, you should two-box (or am I wrong?)
In Prisoner, before knowing the opponent's choice, you should cooperate. After knowing the opponent's choice, you should defect (or am I wrong?).
If I'm right in the above two cases, doesn't Omega look more like the "after knowing" situations above? If so, then I must be wrong about the above two cases...
I want to be someone who in situation Y does X, but when Y&Z happens, I don't necessarily want to do X. Here, Z is the extra information that I lost (in Omega), the opponent has chosen (in Prisoner) or that both boxes have money in them (in Newcomb). What am I missing?
No - in the prisoners' dilemma, you should always defect (presuming the payoff matrix represents utility), unless you can somehow collectively pre-commit to co-operating, or it is iterative. This distinction you're thinking of only applies when reverse causation comes into play.
You know, if Omega is truly doing a full simulation of my cognitive algorithm, then it seems my interactions with him should be dominated by my desire for him to stop it, since he is effectively creating and murdering copies of me.
The decision doesn't need to be read off from a straightforward simulation, it can be an on-demand, so to say, reconstruction of the outcome from the counterfactual. I believe it should be possible to calculate just your decision, without constructing a morally significant computation. Knowing your decision may be as simple as checking whether you adhere a certain decision theory.
but...if you're the emulated you...you're going to die after you give him/don't give him the money.
It's not just about the USB sticks -- to me that seems inert. But if he's running you off those USB sticks for (let's say) a few hours every day, then you could (in fact there is a 1000/1001 chance that you will) wake up tomorrow morning and find yourself running from one of those drives, and know that there is a clear horizon of a few hours on the subjective experiences you can anticipate. This is a prospect which I, at least, would find terrifying.
My two bits: Omega's request is unreasonable.
Precommitting is something that you can only do before the coin is flipped. That's what the "pre" means. Omega's game rewards a precommitment, but Omega is asking for a commitment.
Precommitting is a rational thing to do because before the coin toss, the result is unknown and unknowable, even by Omega (I assume that's what "fair coin" means). This is a completely different course of action than committing after the coin toss is known! The utility computation for precommitment is not and should not be the same as the one for commitment.
In the example, you have access to information that pre-you doesn't (the outcome of the flip). If rationalists are supposed to update on new information, then it is irrational for you to behave like pre-you.
By definition, pre-you only has access to the coin's probability distribution, while you have access to the result of the coin flip. Surely you don't mean to say that's the same thing?
From the perspective of a non-superintelligence, Omega's prediction abilities are indistinguishable from magic. Human beings can't tell what they "imply." Trying to figure out the implications with a primate brain will only get you into a paradox like claiming a fact is the same as a probability distribution. All we can reasonably do is stipulate Omega's abilities needed to make the problem work and no further.
Hi,
My name is Omega. You may have heard of me.
Anyway, I have just tossed a fair coin, and given that the coin came up tails, I'm gonna have to ask each of you to give me $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But see, if the coin came up heads instead of tails, I'd have given you each $10000, but only to those that would agree to give me $100 if the coin came up tails.
You forgot to add that we have sufficient reason to believe everything you say.
I don't believe you.
Normally, you can assume your thought processes are uncorrelated with whats out there. Newcomb-like problems however, do have the state of the outside universe correlated with your actual thoughts, and this is what throws people off.
If you are unsure if the state of the universe is X or Y (say with p = 1/2 for simplicity), and we can chose either option A or B, we can calculate the expected utility of choosing A vs B by taking 1/2u(A,X)+1/2u(A,Y) and comparing it to 1/2u(B,X)+1/2u(B,Y).
In a newcomb-like problem, where the state of the experiment is actually dependent on your choice, the expected utility comparison should now be ~1u(A,X)+~0u(A,Y) vs ~0u(B,X)+~1u(B,Y).
In this case, it boils down to "Is u(A,X) > u(B,Y)?".
It is not enough for Omega to have a decent record of getting it right, since you could probably do pretty well by reading peoples comments and guessing based on that.
If Omega made its prediction solely based on a comment you made on LessWrong, you should expect that if you choose A the universe will be in the same state as if you choose b- knowing your ultimate decision doesn't tell you anything, since the only relevant evidence is what you said a month ago.
If, however, Omega actually simulates your thought process in sufficient detail to know for sure which choice you made, knowing that you ultimately decide to pick A is strong evidence that omega has set up X, and if you choose B, you better expect to see Y.
The reason that the answer changes is that the state of the box actually does depend on the thoughts themselves- it's just that you thought the same thoughts when omega was simulating you before filling the boxes/flipping the coin.
If you aren't sure whether you're just Omega's simulation, you better one box/pay omega. If we're talking about a wannabe Omega that just makes decent predictions based off comments, then you defect (though if you actually expect a situation like this to come up, you argue that you won't)
Omega's actions depend only on your decision (action), or in this case counterfactual decision, not on your thoughts or the algorithm you use to reach the decision. The action of course depends on your thoughts, but that's the usual case. You may move several steps back, seeking the ultimate cause, but that's pretty futile.
I'm very torn on this problem. Every time I think I've got it figured out and start typing out my reasons why, I change my mind, and throw away my 6+ paragraph explanation and start over, arguing the opposite case, only to change my mind again.
I think the problem has to do with strong conflicts between my rational arguments and my intuition. This problem is a much more interesting koan for me than one hand clapping, or tree in the forest.
There is a caveat: if you are an agent who is constructed to live in the world where Omega tossed its coin to come out tails, so that the state space for which your utility function and prior are defined doesn't contain the areas corresponding to the coin coming up heads, you don't need to give up $100. You only give up $100 as a tribute to the part of your morality specified on the counterfactual area of the state space.
I really fail to see why you're all so fascinated by Newcomb-like problems. When you break causality, all logic based on causality doesn't function any more. If you try to model it mathematically, you will get inconsistent model always.
There's no need to break causality. You are a being implemented in chaotic wetware. However, there's no reason to think we couldn't have rational agents implemented in much more predictable form, as python routines for example, so that any being with superior computation power could simply inspect the source and determine what the output would be.
In such a case, Newcomb-like problems would arise, perfectly lawfully, under normal physics.
You cannot do that without breaking Rice's theorem. If you assume you can find out the answer from someone else's source code -> instant contradiction.
You cannot work around Rice's theorem or around causality by specifying 50.5% accuracy independently of modeled system, any accuracy higher than 50%+epsilon is equivalent to indefinitely good accuracy by repeatedly predicting (standard cryptographic result), and 50%+epsilon doesn't cause the paradox.
Give me one serious math model of Newcomb-like problems where the paradox emerges while preserving causality. Here are some examples. Then you model it, you either get trivial solution to one-box, or causality break, or omega loses. * You decide first what you would do in every situation, omega decides second, and now you only implement your initial decision table and are not allowed to switch. Game theory says you should implement one-boxing. * You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. Game theory says you should precommit to one-box, then implement two-boxing, omega loses. * You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. If omega always decides correctly, then he bases his decision on your switch, which either turns it into model #1 (you cannot really switch, precommitment is binding), or breaks causality.
Rice's theorem says you can't predict every possible algorithm in general. Plenty of particular algorithms can be predictable. If you're running on a classical computer and Omega has a copy of you, you are perfectly predictable.
And all of your choices are just as real as they ever were, see the OB sequence on free will (I think someone referred to it already).
And the argument that omega just needs predictive power of 50.5% to cause the paradox only works if it works against ANY arbitrary algorithm. Having that power against any arbitrary algorithm breaks Rice's Theorem, having that power (or even 100%) against just limited subset of algorithms doesn't cause the paradox.
If you take strict decision tree precommitment interpretation, then you fix causality. You decide first, omega decides second, game theory says one-box, problem solved.
Decision tree precommitment is never a problem in game theory, as precommitment of the entire tree commutes with decisions by other agents:
are identical, as B cannot decide based on f. So the changing your mind problem never occurs.
With omega:
I don't think the paradox exist in any plausible mathematization of the problem. It looks to me like another of those philosophical problems that exist because of sloppiness of natural language and very little more, I'm just surprised that OB/LW crowd cares about this one and not about others. OK, I admit I really enjoyed it the first time I saw it but just as something fun, nothing more than that.
In fact, Newcomb-like problems fall naturally out of any ability to simulate and predict the actions of other agents. Omega as described is essentially the limit as predictive power goes to infinity.
If we define an imperfect predictor as a perfect predictor plus noise, i.e. produces the correct prediction with probability p regardless of the cognition algorithm it's trying to predict, then Newcomb-like problems are very robust to imperfect prediction: for any p > .5 there is some payoff ratio great enough to preserve the paradox, and the required ratio goes down as the prediction improves. e.g. if 1-boxing gets 100 utilons and 2-boxing gets 1 utilon, then the predictor only needs to be more than 50.5% accurate. So the limit in that direction favors 1-boxing.
What other direction could there be? If the prediction accuracy depends on the algorithm-to-be-predicted (as it would in the real world), then you could try to be an algorithm that is mispredicted in your favor... but a misprediction in your favor can only occur if you actually 2-box, so it only takes a modicum of accuracy before a 1-boxer who tries to be predictable is better off than a 2-boxer who tries to be unpredictable.
I can't see any other way for the limit to turn out.
If you have two agents trying to precommit not to be blackmailed by each other / precommit not to pay attention to the others precommitment, then any attempt to take a limit of this Newcomblike problem does depend on how you approach the limit. (I don't know how to solve this problem.)
The value(s) for which the limit is being taken here is unidirectional predictive power, which is loosely a function of the difference in intelligence between the two agents; intuitively, I think a case could be made that (assuming ideal rationality) the total accuracy of mutual behavior prediction between two agents is conserved in some fashion, that doubling the predictive power of one unavoidably would roughly halve the predictive power of the other. Omega represents an entity with a delta-g so large vs. us that predictive power is essentially completely one-sided.
From that basis, allowing the unidirectional predictive power of both agents to go to infinity is probably inherently ill-defined and there's no reason to expect the problem to have a solution.
Whaddaya mean humans are rational agents with access to genuine randomness? That's what we're arguing about in the first place!
Perhaps Omega is entangled with your brain such that in all the worlds in which you would choose to one-box, he would predict that you one-box, and all the worlds in which you would choose to two-box, he would predict that you two-box?
In the original formulation, if Omega expects you to flip a coin, he leaves box B empty.
You wouldn't know this with certainty* because it wouldn't be true.
(*unless you were delusional)
The guy next door is on roughly your mental level. Thus, the guy next door can't predict your actions perfectly, because he can't run a perfect simulation of your mind that's faster than you. He doesn't have the capacity.
And he certainly doesn't have the capacity to simulate the environment, including other people, while doing so.
Humans may or may not generally have access to genuine randomness.
It's as yet unknown whether we even have run on quantum randomness; and its also unprovable that quantum randomness is actually genuine randomness, and not just based on effects we don't yet understand, as so many other types of randomness have been.
You're not taking this in the least convenient possible world. Surely it's not impossible in principle that your neighbor can simulate you and your environment. Perhaps your neighbor is superintelligent?
It's ALSO not impossible in principle in the real world. A superintelligent entity could, in principle, perfectly predict my actions. Remember, in the Least Convenient Possible World quantum "randomness" isn't random.
As such, this ISN'T a fundamental difference between humans and "such beings". Which was all I set out to demonstrate.
I was using the "most plausible world" on the basis that it seemed pretty clear that that was the one Roko intended. (Where your neighbour isn't in fact Yahweh in disguise). EDIT: Probably should specify worlds for things in this kind of environment. Thanks, the critical environment here is helping me think about how I think/argue.
On second thoughts, since many clever philosophers spend careers on these problems, I may be missing something.
Nah, they just need something to talk about.
I find the problem interesting, so I'll try to explain why I find it interesting.
So there are these blogs called Overcoming Bias and Less Wrong, and the people posting on it seem like very smart people, and they say very reasonable things. They offer to teach how to become rational, in the sense of "winning more often". I want to win more often too, so I read the blogs.
Now a lot of what these people are saying sounds very reasonable, but it's also clear that the people saying these things are much smarter than me; so much so that although their conclusions sound very reasonable, I can't always follow all the arguments or steps used to reach those conclusions. As part of my rationalist training, I try to notice when I can follow the steps to a conclusion, and when I can't, and remember which conclusions I believe in because I fully understand it, and which conclusions I am "tentatively believing in" because someone smart said it, and I'm just taking their word for it for now.
So now Vladimir Nesov presents this puzzle, and I realize that I must not have understood one of the conclusions (or I did understand them, and the smart people were mistaken), because it sounds like if I were to follow the advice of this blog, I'd be doing something really stupid (depending on how you answered VN's problem, the stupid thing is either "wasting $100" or "wasting $4950").
So how do I reconcile this with everything I've learned on this blog?
Think of most of the blog as a textbook, with VN's post being an "exercise to the reader" or a "homework problem".
The primary reason for resolving Newcomb-like problems is to explore the fundamental limitations of decision theories.
It sounds like you are still confused about free will. See Righting a Wrong Question, Possibility and Could-ness, and Daniel Dennett's lecture here.
I think I'm not confused about free will, and that the links I gave should help to resolve most of the confusion. Maybe you should write a blog post/LW article where you formulate the nature of your confusion (if you still have it after reading the relevant material), I'll respond to that.
Not really - all that is neccessary is that Omega is a sufficiently accurate predictor that the payoff matrix, taking this accuracy into question, still amounts to a win for the given choice. There is no need to be a perfect predictor. And if an imperfect, 99.999% predictor violates free will, then it's clearly a lost cause anyway (I can predict with similar precision many behaviours about people based on no more evidence than their behaviour and speech, never mind godlike brain introspection) Do you have no "choice" in deciding to come to work tomorrow, if I predict based on your record that you're 99.99% reliable? Where is the cut-off that free will gets lost?
Chances are I can predict such a response too, and so won't tell you of my prediction (or tell you in such a way that you will be more likely to attend: eg. "I've a $50 bet you'll attend tomorrow. Be there and I'll split it 50:50"). It doesn't change the fact that in this particular instance I can fortell the future with a high degree of accuracy. Why then would it violate free will if Omega could predict your accuracy in this different situation (one where he's also able to predict the effects of him telling you) to a similar precision?
Then take my bet situation. I announce your attendance, and cut you in with a $25 stake in attendance. I don't think it would be unusual to find someone who would indeed appear 99.99% of the time - does that mean that person has no free will?
People are highly, though not perfectly, predictable under a large number of situations. Revealing knowledge about the prediction complicates things by adding feedback to the system, but there are lots of cases where it still doesn't change matters much (or even increases predictability). There are obviously some situations where this doesn't happen, but for Newcombe's paradox, all that is needed is a predictor for the particular situation described, not any general situation. (In fact Newcombe's paradox is equally broken by a similar revelation of knowledge. If Omega were to reveal its prediction before the boxes are chosen, a person determined to do the opposite of that prediction opens it up to a simple Epimenides paradox.)
I would one-box on Newcombe, and I believe I would give the $100 here as well (assuming I believed Omega).
With Newcombe, if I want to win, my optimal strategy is to mimic as closely as possible the type of person Omega would predict would take one box. However, I have no way of knowing what would fool Omega: indeed if it is a sufficiently good predictor there may be no such way. Clearly then the way to be "as close as possible" to a one-boxer is to be a one-boxer. A person seeking to optimise their returns will be a person who wants their response to such stimulus to be "take one box". I do want to win, so I do want my response to be that, so it is: I'm capable of locking my decisions (making promises) in ways that forgo short-term gain for longer term benefit.
The situation here is the same, even though I have already lost. It is beneficial for me to be that type of person in general (obscured by the fact that the situation is so unlikely to occur). Were I not the type of person who made the decision to pay out on loss, I would be the type of person that lost $10000 in an equally unlikely circumstance. Locking that response in now as a general response to such occurrances means I'm more likely to benefit than those who don't.
Well, the other way to look at it is "What action leads me to win?" in the Newcomb problem, one-boxing wins, so you and I are in agreement there.
But in this problem, not-giving-away-$100 wins. Sure, I want to be the "type of person who one boxes", but why do I want to be that person? Because I want to win. Being that type of person in this problem actually makes you lose.
The problem states that this is a one-shot bet, and that after you do or don't give Omega the $100, he flies away from this galaxy and will never interact with you again. So why give him the $100? It won't make you win in the long term.
Yes, but Omega isn't really here yet, and you, Nebu, deciding right now that you will give him $100 does make you win, since it gives you a shot at $10000.
Right, so if a normal person offered me the bet (and assuming I could somehow know it was a fair coin) then yes, I would accept the bet.
If it was Omega instead of a normal person offering the bet, we run into some problems...
But if Omega doesn't actually offer the bet, and just does what is described by Vladimir Nesov, then I wouldn't give him the $100. [1]
In other words, I do different things in different situations.
Edit 1: (Or maybe I would. I haven't figured it out yet.)
The problem only asks about what you would do in the failure case, and I think this obscures the fact that the relevant decision point is right now. If you would refuse to pay, that means that you are the type of person who would not have won had the coin flip turned out differently, either because you haven't considered the matter (and luckily turn out to be in the situation where your choice worked out better), or because you would renege on such a commitment when it occurred in reality.
However at this point, the coin flip hasn't been made. The globally optimal person to be right now is one that does precommit and doesn't renege. This person will come out behind in the hypothetical case as it requires we lock ourselves into the bad choice for that situation, but by being a person who would act "irrationally" at that point, they will outperform a non-committer/reneger on average.
What if there is no "on average", if the choice to give away the $100 is the only choice you are given in your life? There is no value in being the kind of person who globally optimizes because of the expectation to win on average. You only make this choice because it's what you are, not because you expect the reality on average to be the way you want it to be.
From my perspective now, I expect the reality to be the winning case 50% of the time because we are told this as part of the question: Omega is trustworthy and said it tossed a fair coin. In the possible futures where such an event could happen, 50% of the time my strategy would have paid off to a greater degree than it would lose the other 50% of the time. If omega did not toss a fair coin, then the situation is different, and my choice would be too.
There is no value in being such a person if they happen to lose, but that's like saying there's no value in being a person who avoids bets that lose on average by only posing the 1 in several million time they would have won the lottery. On average they'll come out ahead, just not in the specific situation that was described.
I think my answer would be "I would have agreed, had you asked me when the coin chances were .5 and .5. Now that they're 1 and 0, I have no reason to agree."
Seriously, why stick with an agreement you never made? Besides, if Omega can predict me this well he knows how the coin will come up and how I'll react. Why then, should I try to act otherwise. Somehow, I think I just don't get it.
Whether I give Omega the $100 depends entirely on whether there will be multiple iterations of coin-flipping. If there will be multiple iterations, giving Omega the $100 is indeed winning, just like buying a financial instrument that increases in value is winning.
No, there are no iterations. Omega flies away from your galaxy, right after finishing the transaction. (Added to P.S.)
In that case, I'd hate to disappoint Omega, but there's no incentive for me to give up my $100. A utility of 0 is better than a negative utility, and if the coin-flip is deterministic, I won't be serving the interests of my alternate-universe self. Why would I choose otherwise?
Would you prefer to choose otherwise if you considered the deal before the actual coin toss, and arrange the precommitment to that end?
Yes, then, following the utility function you specified, I would gladly risk $100 for an even chance at $10000. Since Omega's omniscient, I'd be honest about it, too, and cough up the money if I lost.
If it's rational to do this when Omega asks you in advance, isn't it also rational to make such a commitment right now? Whether you make the commitment in response to Omega's notification, or on a whim when considering the thought experiment in response to a blog post makes no difference to the payoff. If you now commit to a "if this exact situation comes up, I will commit to paying the $100 if I lose the coinflip", and p(x) is the probability of this situation occurring, you will achieve a net gain of $4950*p(x) over a non-committer (a very small number admittedly given that p(x) is tiny, but for the sake of the thought experiment all that matters is that it's positive.)
Given that someone who makes such a precommitment comes out ahead of someone who doesn't - shouldn't you make such a commitment right now? Extend this and make a precommitment to always make the decision to perform the action that would maximise your average returns in all such newcombelike situations and you're going to come off even better on average.
No, I will not precommit to giving up my $100 for cases where Omega demands the money after the coin flip has occurred. There is no incentive to precommit in those cases, because the outcome is already against me and there's not a chance that it "would" go in my favour.
At that point, it's no longer a precommittal - it's how you face the consequences of your decision whether to precommit or not.
Note that the hypothetical loss case presented in the post is not in fact the decision point - that point is when you first consider the matter, which is exactly what you are doing right now. If you would really change your answer after considering the matter, then having now done so, have you changed it?
If you want to obtain the advantage of someone who makes such a precommittal (and sticks to it), you must be someone who would do so. If you are not such a person (and given your answer, you are not) it is advantageous to change yourself to be such a person, by making that precommitment (or better, a generalised "I will always take the path would have maximised returns across the distribution of counterfactual outcomes in Newcomblike situations") immediately.
Such commitments change the dynamics of many such thought experiments, but usually they require that that commitment be known to the other person, and enforced some way (The way to win at Chicken is to throw your steering wheel out the window). Here though, Omega's knowledge of us removes the need to explicit announcement, and it is in our own interests to be self-enforcing (or rather we wish to reliably enforce the decision on our future selves), or we will not receive the benefit. For that reason, a silent decision is as effective as having a conversation with Omega and telling it how we decide.
Explicitly announcing our decision thus only has an effect insofar as it keeps your future self honest. Eg. if you know you wouldn't keep to a decision idly arrived at, but value your word such that you would stick to doing what you said you would despite its irrationality in that case, then it is currently in your interest to give your word. It's just as much in your interest to give your word now though - make some public promise that you would keep. Alternatively if you have sufficient mechanisms in your mind to commit to such future irrational behaviour without a formal promise, it becomes unneccessary.
Maybe in thought-experiment-world. But if there's a significant chance that you'll misidentify a con man as Omega, then this tendency makes you lose on average.
Sure - all bets are off if you aren't absolutely sure Omega is trustworthy.
I think this is a large part of the reason why the intuitive answer we jump to is rejection. Being told we believe a being making such extraordinary claims is different to actually believing them (especially when the claims may have unpleasant implications to our beliefs about ourselves), so have a tendency to consider the problem with the implicit doubt we have for everyday interactions lurking in our minds.
So after you observe the coin toss, and find yourself in a position where you've lost, you'll give Omega your money? Why would you? It won't ever reciprocate, and it won't enforce the deal, its only enforcement are those $10000 that you know got away anyway, because you didn't win the coin toss.
Yes, I'll give Omega the money, because if I'm going to refuse to give Omega the money after the coin toss occurs, Omega knows ahead of time on account of his omniscience. If I had won, Omega could look at me and say, "You get no money, because I know you wouldn't have really given me the $100 if you'd lost. Your pre-commitment wasn't genuine."
My answer to this is that integrity is a virtue, and breaking one's promises reduces one's integrity. And being a person with integrity is vital to the good life.
Then I repeat the question with MBlume's corrections, to make the problem less convenient. Would you still follow up and murder 15 people, to preserve your personal integrity? It's not a question of values, it's a question of decision theory.
This thread assumes a precommitment. I would not precommit to murder.
I'm not sure what your point is here.
The point is that the distinction between $0.02 and a trillion lives is irrelevant to the discussion, which is about the structure of preference order assigned to actions, whatever your values are. If you are determined to pay off Omega, the reason for that must be in your decision algorithm, not in an exquisite balance between $100, personal integrity, and murder. If you are willing to carry the deal through (note that there isn't even any deal, only your premeditated decision), the reason for that must lie elsewhere, not in the value of personal integrity.
Precommitting should be, as someone already said, signing a paper with a third party agreeing to give them $1000 in case you fail to give the $100 to Omega. Precommitment means you have no other option. You can't say that you both precommitted to give the $100 AND refused to do it when presented with the case.
Which means, if Omega presents you with the scenario before the coin toss, you precommit (by signing the contract with the third party). If Omega presents you with the scenario after the coin toss AND also tells you it has already come up tails - you haven't precommited, therefore you shouldn't give it $100.
EDIT: Also, some people objected to not giving the $100, because they might be the emulation which Omega uses to predict whether you'd really give money. If you were an emulation, then you would remember precommitting in expectation to get $10,000 with a 50% chance. It makes no sense for Omega to emulate you in a scenario where you don't get a chance to precommit.
That level of precomitting is only neccessary if you are unable to trust yourself to carry through with a self-imposed precommitment. If you are capable of this, you can decide now to act irrationally to certain future decisions in order to benefit to a greater degree than someone who can't. If the temptation to go back on your self-promise is too great in the failure case, then you would have lost in the win case - you are simply a fortunate loser who found out the flaw in his promise in the case where being flawed was beneficial. It doesn't change the fact that being capable of this decision would be a better strategy on average. Making yourself conditionally less rational can actually be a rational decision, and so the ability to do so can be a strength worth acquiring.
Ultimately the problem is the same as that of an ultimatum (eg. MAD). We want the other party to believe we will carry through even if it would be clearly irrational to do so at that point. As your opponent becomes better and better at predicting, you must become closer and closer to being someone who would make the irrational decision. When your opponent is sufficiently good (or you have insufficient knowledge as to how they are predicting), the only way to be sure is to be someone who would actually do it.
Okay, I agree that this level of precomitting is not necessary. But if the deal is really a one-time offer, then, when presented with the case of the coin already having come up tails, you can no longer ever benefit from being the sort of person who would precommit. Since you will never again be presented with a newcomb-like scenario, then you will have no benefit from being the precommiting type. Therefore you shouldn't give the $100.
If, on the other hand, you still expect that you can encounter some other Omega-like thing which will present you with such a scenario, doesn't this make the deal repeatable, which is not how the question was formulated?
In the repeatable scenario I believe, unlike Vladimir, that a real difference exists. Whatever decision process you use to decide not to pay $100 in one round, you can predict with high probability that that same process will operate in future rounds as well, leading to a total gain to you of about $0. On the other hand, you know that if your current decision process leads you to giving $100 in this case, then with high probability that same process will operate in future rounds, leading to a total gain to you of about $4950 x expected future rounds. Therefore, if you place a higher confidence in your ability to predict your future actions from your current ones than you do in your own reasoning process, you should give the $100 up. This makes the problem rather similar to the original Newcomb's problem, in that you assign higher probability that your reasoning is wrong if it causes you to two-box than you do to any reasoning which leads you to two-box.
This is a self-deception technique. If you think it's morally OK to self-deceive your future self for your current selfish ends, then by all means go ahead. Also, it looks like violent means of precommitment should actually be considered immoral, on par with forcing some other person to do your bidding by hiring a killer to kill them if they don't comply.
In the Newcomb's problem, it actually is in your self-interest to one-box. Not so in this problem.
I am fairly sure that it isn't, but demonstrating so would require another maths-laden article, which I anticipate would be received similarly to my last. I will however email you my entire reasoning if you so wish (you will have to wait several days while I brush up on the logical concept of common knowledge). (I don't know how to encode a ) in a link, so please add one to the end.)
Common knowledge (I used the %29 ASCII code for ")").
I'm going to write up my new position on this topic. Nonetheless I think it should be possible to discuss the question in a more concise form, since I think the problem is that of communication, not rigor. You deceive your future self, that's the whole point of the comment above, make it believe that it wants to make an action that it actually doesn't. The only disagreement position that I expect is saying that no, the future self actually wants to follow that action.
I think the problem with your article wasn't that it was math-laden, but that you didn't introduce things in sufficient detail to follow along, and to see the motivation behind the math.
To be perfectly honest, your last sentence is also my feeling. I should at the least have talked more about the key equation. But the article was already long, I was unsure as to how it would be received, and I spent too little time revising it (this is a persistent problem for me). If I were to write it again now, it would have been closer in style to the thread between you and me there.
If you intend to write another post, then I am happy to wait until then to introduce the ideas I have in mind, and I will try hard to do so in a manner that won't alienate everyone.
In a repeatable deal your action influences the conditions in the next rounds. Even if you defect in this round, you may still cooperate in the next rounds, Omegas aren't looking back at how you decided in the past, and don't punish you by not offering the deals. Your success in the following rounds (from your current point of view) depends on whether you manage to precommit to the future encounters, not on what you do now.
If you think that through and decide that way, then your precommitting method didn't work. The idea is that you must somehow now prevent your future self from behaving rationally in that situation - if they do, they will perform exactly the thought process you describe. The method of doing so, whether making a public promise (and valuing your spoken word more than $100), hiring a hitman to kill you if you renege or just having the capability of reliably convincing yourself to do so (effectively valuing keeping faith with your self-promise more than $100) doesn't matter so long as it is effective. If merely deciding now is effective, then that is all that's needed.
If you do then decide to take the rational course in the losing coinflip case, it just means you were wrong by definition about your commitment being effective. Luckily in this one case, you found it out in the loss case rather than the win case. Had you won the coin flip, you would have found yourself with nothing though.
Suppose Omega gives you the same choice, but says that if a head had come up, it would have killed you, but only if you {would have refused|will refuse} to give it your lousy $100 {if the coin had come up heads|given that the coin has come up heads}. Not sure what the correct tense is, here.
I believe that I would keep the $100 in your problem, but give it up in mine.
ETA: Can you clarify your postscript? Presumably you don't want the knowledge about the distribution of coin-flip states across future Everett branches to be available for the purposes of the expected utility calculation?
I'm trying to set up a sufficiently inconvenient possible world by introducing additional assumptions. The one about MWI stops the excuse of there being other real you in the other MWI branches who do receive the $10000. Not allowed.
How do you peak the threshold, decide that [$10000] < [decision threshold] < [your life]?
You've actually made it an easier problem for me, though, because I regard my alternate selves as other people.
How do you peak the threshold, decide that [$10000] < [decision threshold] < [your life]?
If it were possible for me to make a deal with my alternate self by which I get a few thousand dollars, I would obviously surrender my $100. As it isn't possible, I see little reason to give someone otherwise destined to be forever causally isolated from me $10000 at the cost of $100. I wouldn't keep $100 if it meant he lost $10000, either. I probably would keep the $100 if they lost less than $100. If my alternate self stood to gain, say, a million dollars, but nothing if I kept my $100, then I probably would give it up. But that would be as a whimsy, something to think about and feel good. But the benefit to me of that whimsy would have to be worth more than $100.
The pattern behind my choices is that the pain experienced by my alternate self (who, recall, I consider a different person) in any of these cases is never more than $100. I think this is the most we can expect, on average, of other intelligent beings: that they will not inflict a large loss for a small gain. Why not steal, in that case? Because there is, in fact, no such thing as total future causal isolation.
There is no alternative self. None at all. The alternative may be impossible according to the laws of physics. It is only present in your imperfect model of the world. You can't trade with a fiction, and you shouldn't emphasize with a fiction. What you decide, you decide in this our real world. You decide that it is right to make a sacrifice, according to your preferences that only live in your model of the world, but speak about the reality.
I think that this is a critical point, worthy of a blog post of its own. Impossible possible worlds are a confusion.
The inclination to trade with fiction seems like a serious problem within this community.
I've misunderstood you to an extent, then.
My preferences don't involve me sacrificing unless someone can get hurt. It doesn't matter whether that person exists in another Everett branch, within Omega or in another part of the Tegmark ensemble, but there must be a someone. I'll play symmetrist with everyone else (which is, in a nutshell, what I said in my comment above) but not with myself. You seem to want a person that is me, but minus the "existence" property. I don't think that is a coherent concept.
OK, suppose that Omega came along right now and said to me "I have determined that if you could be persuaded that your actions would have no consequence, and then given the problem you are currently discussing, you would in every case keep $100. Therefore I will torture you endlessly." I would not see this as proof of my irrationality (in the sense of hopelessly failing to achieve my preferences). I don't think that such a sequence of events is germane to the problem as you see it, but I also don't see how it is not germane.
We're assuming Omega is trustworthy? I'd give it the $100, of course.
Had the coin come up differently, Omega might have explained the secrets of friendly artificial general intelligence. However, he now asks that you murder 15 people.
Omega remains completely trustworthy, if a bit sick.
Raising the stakes in this way does not work, because of the issue described in Ethical Injunctions: it is less likely that Omega has presented you with this choice, than that you have gone insane.
That may be true, but it's still a dodge. Conditional on not being insane, what's your answer?
Additionally, I don't see why Omega asking you to give it 100 dollars vs 15 human lives necessarily crosses the threshold of "more likely that I'm just a nutbar". I don't expect to talk to Omega anytime soon...
So imagine yourself in the most inconvenient possible world where Omega is a known feature of the environment and has long been seen to follow through on promises of this type; it does not particularly occur to you or anyone that believing this fact makes you insane.
When I phrase it that way - imagine myself in a world full of other people confronted by similar Omega-induced dilemmas - I suddenly find that I feel substantially less uncomfortable; indicating that some of what I thought was pure ethical constraint is actually social ethical constraint. Still, it may function to the same self-protective effect as ethical constraint.
To add to the comments below, if you're going to take this route, you might as well have already decided that encountering Omega at all is less likely than that you have gone insane.
Completely agreed, a major problem in any realistic application of such scenarios.
I'm afraid I don't follow.
Ouch.
For some reason, raising the stakes in these hypotheticals to the point of actual pain has become reflex for me. I'm not sure if it's to help train my emotions to be able to make the right choices in horrible circumstances, or just my years in the Bardic Conspiracy looking for an outlet.
So from my and Omega's perspective this coin is random and my behavior is predictable. Amusing. My question: What if Omega says "due to quirks in your neurology, had I requested it, you would have pre-committed to bet $100 against $46.32. As it happens, you lost anyway, but you would have taken an unfavorable deal. Would you pay then?
Nope. I don't care what quirks in my neurology do - I don't care what answer the material calculator returns, only the answer to 2 + 2 = ?
The coin toss may be known to Omega and predicted in advance, it only needs to initially have 50/50 odds to you for the expected gain calculation to hold. When Omega tells you about the coin, it communicates to you its knowledge about the toss, about an independent variable of initial 50/50 odds. For example, Omega may tell you that it hasn't tossed the coin yet, it'll do so only a thousand years from now, but it predicted that the coin will come up tails, so it asks you for your $100.
That's just like playing "Eeny, meeny, miny, moe" to determine who's 'it'. Once you figure out if there's an even or odd number of words, you know the answer, and it isn't random to you anymore. This may be great as a kid choosing who gets a cookie (wow! I win again!), but you're no longer talking about something that can go either way.
For a random output of a known function, you still need a random input.
This requires though that Omega have decided to make the bet in a fashion which exhibited no dependency on its advance knowledge of the coin.
This is a big issue which I unsucessfully tried to address in my non-existing 6+ paragraph explanation. Why the heck is Omega making bets if he can already predict everything anyway?
That said, it's not clear that when Omega offers you a bet, you should automatically refuse it under the assumption that Omega is trying to "beat" you. It seems like Omega doesn't really mind giving away money (pretty reasonable for an omniscient entity), since he seems to be willing to leave boxes with millions of dollars in them just lying around.
What is Omega's purpose is entirely unknown. Maybe he wants you to win these bets. If you're a rational person who "wants to win", I think you can just "not worry" about what Omega's intents are, and figure out what sequence of actions maximizes your utility (which in these examples always seems to directly translate into maximizing the amount of money you get).
Can you please explain the reasoning behind this? Given all of the restrictions mentioned (no iterations, no possible benefit to this self) I can't see any reason to part with my hard earned cash. My "gut" says "Hell no!" but I'm curious to see if I'm missing something.
I work on AI. In particular, on decision systems stable under self-modification. Any agent who does not give the $100 in situations like this will self-modify to give $100 in situations like this. I don't spend a whole lot of time thinking about decision theories that are unstable under reflection. QED.
Even considering situations like this and having special cases for them sounds like it would add a bit much cruft to the system.
Do you have a working AI that I could look at to see how this would work?
If you need special cases, your decision theory is not consistent under reflection. In other words, it should simply always do the thing that it would precommit to doing, because, as MBlume put it, the decision theory is formulated in such fashion that "What would you precommit to?" and "What will you do?" work out to be one and the same question.
But this is precisely what humans don't do, because we respond to a "near" situation differently than a "far" one. Your advance prediction of your decision is untrustworthy unless you can successfully simulate the real future environment in your mind with sufficient sensory detail to invoke "near" reasoning. Otherwise, you will fail to reach a consistent decision in the actual situation.
Unless of course, In the actual situation, you're projecting back, "What would I have decided in advance to do had I thought about this in advance?" -- and you successfully mitigate all priming effects and situationally-motivated reasoning.
Or to put all of the above in short, common-wisdom form: "that's easy for you to say NOW..." ;-)
There are various intuition pumps to explain the answer.
The simplest is to imagine that a moment from now, Omega walks up to you and says "I'm sorry, I would have given you $10000, except I simulated what would happen if I asked you for $100 and you refused". In that case, you would certainly wish you had been the sort of person to give up the $100.
Which means that right now, with both scenarios equally probable, you should want to be the sort of person who will give up the $100, since if you are that sort of person, there's half a chance you'll get $10000.
If you want to be the sort of person who'll do X given Y, then when Y turns up, you'd better bloody well do X.
Is that an acceptable correction?
Well, with a being like Omega running around, the two become more or less identical.
If we're going to invent someone who can read thoughts perfectly, we may as well invent someone who can conceal thoughts perfectly.
Anyway, there aren't any beings like Omega running around to my knowledge. If you think that concealing motivations is harder than I think, and that the only way to make another human think you're a certain way is to be that way, say that.
I think this describes one of the core principles of virtue theory under any ethical system.
I wonder how much it depends upon accidents of human psychology, like our tendency to form habits, and how much of it is definitional (if you don't X when Y, then you're simply not the sort of person who Xes when Y)
Well said. That's a lot of the motivation behind my choice of decision theory in a nutshell.
Thanks, it's good to know I'm on the right track =)
I think this core insight is one of the clearest changes in my thought process since starting to read OB/LW -- I can't imagine myself leaping to "well, I'd hand him $100, of course" a couple years ago.
That's not the situation in question. The scenario laid out by Vladimir_Nesov does not allow for an equal probability of getting $10000 and paying $100. Omega has already flipped the coin, and it's already been decided that I'm on the "losing" side. Join that with the fact that me giving $100 now does not increase the chance of me getting $10000 in the future because there is no repetition.
Perhaps there's something fundamental I'm missing here, but the linearity of events seems pretty clear. If Omega really did calculate that I would give him the $100 then either he miscalculated, or this situation cannot actually occur.
-- EDIT --
There is a third possibility after reading Cameron's reply... If Omega is correct and honest, then I am indeed going to give up the money.
But it's a bit of a trick question, isn't it? I'm going to give up the money because Omega says I'm going to give up the money and everything Omega says is gospel truth. However, if Omega hadn't said that I would give up the money, then I wouldn't of given up the money. Which makes this a bit of an impossible situation.
Assuming the existence of Omega, his intelligence, and his honesty, this scenario is an impossibility.
I don't see this situation is impossible, but I think it's because I've interpreted it differently from you.
First of all, I'll assume that everyone agrees that given a 50/50 bet to win $10'000 versus losing $100, everyone would take the bet. That's a straightforward application of utilitarianism + probability theory = expected utility, right?
So Omega correctly predicts that you would have taken the bet if he had offered it to you (a real no brainer; I too can predict that you would have taken the bet had he offered it).
But he didn't offer it to you. He comes up now, telling you that he predicted that you would accept the bet, and then carried out the bet without asking you (since he already knew you would accept the bet), and it turns out you lost. Now he's asking you to give him $100. He's not predicting that you will give him that number, nor is he demanding or commanding you to give it. He's merely asking. So the question is, do you do it?
I don't think there's any inconsistency in this scenario regardless of whether you decide to give him the money or not, since Omega hasn't told you what his prediction would be (though if we accept that Omega is infallible, then his prediction is obviously exactly whatever you would actually do in that situation).
I feel like a man in an Escher painting, with all these recursive hypothetical mes, hypothetical kuriges, and hypothetical omegas.
I'm saying, go ahead and start by imagining a situation like the one in the problem, except it's all happening in the future -- you don't yet know how the coin will land.
You would want to decide in advance that if the coin came up against you, you would cough up $100.
The ability to precommit in this way gives you an advantage. It gives you half a chance at $10000 you would not otherwise have had.
So it's a shame that in the problem as stated, you don't get to precommit.
But the fact that you don't get advance knowledge shouldn't change anything. You can just decide for yourself, right now, to follow this simple rule:
If there is an action to which my past self would have precommited, given perfect knowledge, and my current preferences, I will take that action.
By adopting this rule, in any problem in which the oppurtunity for precommiting would have given you an advantage, you wind up gaining that advantage anyway.
I'm actually not quite satisfied with it. Probability is in the mind, which makes it difficult to know what I mean by "perfect knowledge". Perfect knowledge would mean I also knew in advance that the coin would come up tails.
I know giving up the $100 is right, I'm just having a hard time figuring out what worlds the agent is summing over, and by what rules.
ETA: I think "if there was a true fact which my past self could have learned, which would have caused him to precommit etc." should do the trick. Gonna have to sleep on that.
ETA2: "What would you do in situation X?" and "What would you like to pre-commit to doing, should you ever encounter situation X?" should, to a rational agent, be one and the same question.
"Perfect knowledge would mean I also knew in advance that the coin would come up tails."
This seems crucial to me.
Given what I know when asked to hand over the $100, I would want to have pre-committed to not pre-committing to hand over the $100 if offered the original bet.
Given what I would know if I were offered the bet before discovering the outcome of the flip I would wish to pre-commit to handing it over.
From which information set I should evaluate this? The information set I am actually at seems the most natural choice, and it also seems to be the one that WINS (at least in this world).
What am I missing?
...and that's an even better way of putting it.
MBlume:
This phrasing sounds about right. Whatever decision-making algorithm you have drawing your decision D when it's in situation X, should also come to the same conditional decision before the situation X appeared, "if(X) then D". If you actually don't give away $100 in situation X, you should also plan to not give away $100 in case of X, before (or irrespective of whether) X happens. Whichever decision is the right one, there should be no inconsistency of this form. This grows harder if you must preserve the whole preference order.
Omega hasn't told you his predictions in the given scenario.
Thank you. Now I grok.
So, if this scenario is logically inconsistent for all values of 'me' then there really is nothing that I can learn about 'me' from this problem. I wish I hadn't thought about it so hard.
And if Omega comes up to me and says "I was going to kill you if you gave me $100. But since I've worked out that you won't, I'll leave you alone." then I'll be damn glad I wouldn't agree.
This really does seem like pointless speculation.
Of course, I live in a world where there is no being like Omega that I know of. If I knew otherwise, and knew something of their properties, I might govern myself differently.
We're not talking Pascal's Wager here, you're not guessing at the behaviour of capricious omnipotent beings. Omega has told you his properties, and is assumed to be trustworthy.
You are stating that. But as far as I can tell Omega is telling me its a capricious omnipotent being. If there is a distinction, I'm not seeing it. Let me break it down for you:
1) Capricious -> I am completely unable to predict its actions. Yes.
2) Omnipotent -> Can do the seemingly impossible. Yes.
So, what's the difference?
It's not capricious in the sense you give: you are capable of predicting some of its actions: because it's assumed Omega is perfectly trustworthy, you can predict with certainty what it will do if it tells you what it will do.
So, if it says it'll give you 10k$ in some condition (say, if you one-box its challenge), you can predict that it'll give it the money if that condition arises.
If it were capricious in the sense of complete inability of being predicted, it might amputate three of your toes and give you a flower garland.
Note that the problem supposes you do have certainty that Omega is trustworthy; I see no way of reaching that epistemological state, but then again I see no way Omega could be omnipotent, either.
On an somewhat unrelated note, why would Omega ask you for 100$ if it had simulated you wouldn't give it the money? Also, why would it do the same if it had simulated you would give it the money? What possible use would an omnipotent agent have for 100$?
If we assume I'm rational, then I'm not going to assume anything about Omega. I'll base my decisions on the given evidence. So far, that appears to be described as being no more and no less than what Omega cares to tell us.
(Disclaimer: I'm going to use the exact language you used, which means I will call you "stupid" in this post. I apologize if this comes off as trollish. I will admit that I am also quite torn about this decision, and I feel quite stupid too.)
No offense, but assuming free will, you are the one who is deciding to actually hand over the $100. The conterfactual isn't the one making the decision. You are. You are in a situation, and there are two possible actions (lose $100 or don't lose $100), and you are choosing to lose $100.
So again, are you sure you are not stupid?
And now I try to calculate what you should treat as being the probability that you're being emulated. Assume that Omega only emulates you if the coin comes up heads.
Suppose you decide beforehand that you are going to give Omega the $100, as you ought to. The expected value of this is $4950, as has been calculated.
Suppose that instead, you decide beforehand that E is the probability you're being emulated assuming you hear that came up tails. You'll still decide to give Omega the $100; therefore, your expected value if you hear that it came up heads is $10,000. Your expected value if you hear that the coin came up tails is -$100(1-E) + $10,000E.
The probability that you hear that the coin comes up tails should be given by P(H) + P(T and ~E) + P(T and E) = 0, P(H) = P(T and ~E), P(T and ~E) = P(T) - P(T and E), P(T and E) = P(E|T) * P(T). Solving these equations, I get P(E|T) = 2, which probably means I've made a mistake somewhere. If not, c'est l'Omega?
This is just the one-shot Prisoner's Dilemma. You being split into two different possible worlds, is just like the two prisoners being taken into two different cells.
Therefore, you should give Omega $100 if and only if you would cooperate in the one-shot PD.
I guess I'm a bit tired of "God was unable to make the show today so the part of Omniscient being will be played by Omega" puzzles, even if in my mind Omega looks amusingly like the Flying Spaghetti Monster.
Particularly in this case where Omega is being explicitly dishonest - Omega is claiming to be either be sufficiently omniscient to predict my actions, or insufficiently omniscient to predict the result of a 'fair' coin, except that the 'fair' coin is explicitly predetermined to always give the same result . . . except . . .
What's the point of using rationalism to think things through logically if you keep placing yourself into illogical philosophical worlds to test the logic?
Coin is not predetermined, and it doesn't matter if Omega has hand-selected every result of the coin toss, as long as we don't have any reason to slide the probability of the result to either direction.
If some guy walked up to you and gave you this spiel, you'd be fully justified in telling him to get lost, or even seeking mental help for him.
The problem assumes Omega to be genuine, and trustworthy.