= 27d567bc2d48d9f40e9ccc20ea370b15)
Regardless of whether the third one is plausible, I suspect Omega would know of some hack that is equally weird and unable to be anticipated.
A sensible thing to consider. You are effectively dealing with an outcome pump, after all; the problem leaves plenty of solution space available, and outcome pumps usually don't produce an answer you'd expect; they instead produce something that matches the criteria even better then anything you were aware of.
In response to
Problem of Optimal False Information
As stated, the question comes down to acting on an opinion you have on an unknown, but within the principles of this problem potentially knowable conclusion about your own utility function. And that is: Which is larger: 1) the amount of positive utility you gain from knowing the most disutile truths that exist for you OR 2) the amount of utility you gain from knowing the most utile falsehoods that exist for you
ALMOST by definition of the word utility, you would choose the truth (white box) if and only if 1) is larger and you would choose the falsehood (black box) only if 2) is larger. I say almost by definition because all answers of the form "I would choose the truth even if it was worse for me" are really statements that the utility you place on the truth is higher than Omega has assumed, which violates the assumption that Omega knows your utility function and speaks truthfully about it.
I say ALMOST by definition because we have to consider the other piece of the puzzle: when I open box 2) there is a machine that "will reprogram your mind." Does this change anything? Well it depends on which utility function Omega is using to make her calculations of my long term utility. Is Omega using my utility function BEFORE the machine reprogram's my mind, or after? Is me after the reprogramming really still me after the reprogramming? I think within the spirit of the problem we must assume that 1) The utility happens to be maximized for both me before the reprogram and me after the reprogram, perhaps my utility function does not change at all in the reprogramming, 2) Omega has correctly included the amount of disutility I would have to the particular programming change, and this is factored in to her calculations so that the proposed falsehood and mind reprogramming do in fact, on net, give the maximum utility I can get from knowing the falsehood PLUS being reprogrammed.
Within these constraints, we find that the "ALMOST" above can be removed if we include the (dis)utility I have for the reprogramming in the calculation. So:
Which is larger: 1) the amount of positive utility you gain from knowing the most disutile truths that exist for you AND being reprogrammed to believe it OR 2) the amount of utility you gain from knowing the most utile falsehoods that exist for you
So ultimately, the question which would we choose is the question above. I think to say anything else is to say "my utility is not my utility," i.e. to contradict yourself.
In my case, I would choose the white box. On reflection, considering the long run, I doubt that there is a falsehood PLUS a reprogramming that I would accept as a combination as more utile than the worst true fact (with no reprogramming to consider) that I would ever expect to get. Certainly, this is the Occam's razor answer, the ceteris paribus answer. GENERALLY, we believe that knowing more is better for us than being wrong. Generally we believe that someone else meddling with our minds has a high disutility to us.
For completeness I think these are straightforward conclusions from "playing fair" in this question, from accepting an Omega as postulated.
1) If Omega assures you the utility for 2) (including the disutility of the reprogramming as experienced by your pre-reprogrammed self) is 1% higher than the utility of 1), then you want to choose 2), to choose the falsehood and the reprogramming. To give any other answer is to presume that Omega is wrong about your utility , which violates the assumptions of the question.
2) If Omega assures you the utility for 2) and 1) are equal, it doesn't matter which one you choose. As much as you might think "all other things being equal I'll choose the truth" you must accept that the value you place on the truth has already been factored in, and the blip-up from choosing the truth will be balanced by some other disutility in a non-truth area. Since you can be pretty sure that the utility you place on the truth is very much unrelated to pain and pleasure and joy and love and so on, you are virtually guaranteeing you will FEEL worse choosing the truth, but that this worse feeling will just barely be almost worth it.
Finally, I tried to play nice within the question. But it is entirely possible, and I would say likely, that there can never be an Omega who could know ahead of time with the kind of detail required, what your future utility would be, at least not in our Universe. Consider just the quantum uncertainties (or future Everett universe splits). It seems most likely that your future net utility covers a broad range of outcomes in different Everett branches. In that case, it seems very likely that there is no one truth that minimizes your utility in all your possible futures, and no one falsehood that maximizes it in all your possible futures. In this case we would have a distribution of utility outcomes from 1) and 2) and it is not clear that we know how to choose between two different distributions. Possibly utility is defined in such a way that it would be the expectation value that "truly" mattered to us, but that puts I think a very serious constraint on utility functions and how we interact with them that I am not sure could be supported.
Quite a detailed analysis, and correct within its assumptions. It is important to know where Omega is getting it's information on your utility function. That said, since Omega implicitly knows everything you know (since it needs to know that in order to also know everything you don't know, and thus to be able to provide the problem at all), it implicitly knows your utility function already. Obviously, accepting a falsehood that perverts your utility function into something counter to your existing utility function just to maximize an easier target would be something of disutility to you as you are at present, and not something that you would accept if you were aware of it. Accordingly, it it a safe assumption that Omega has based its calculations off your utility before accepting the information, and for the purposes of this problem, that is exactly the case. This is your case (2); if a falsehood intrinsically conflicts with your utility function in whatever way, it generates disutility (and thus, is probably suboptimal). If your utility function is inherently hostile to such changes, this presents a limitation on the factset Omega can impose upon you.
That said, your personal answer seems to place rather conservative bounds on the nature of what Omega can do to you. Omega has not presented bounds on it's utilities; instead, it has advised you that they are maximized within fairly broad terms. Simialrly, it has not assured you anything about the relative values of those utilities, but the structure of the problem as Omega presents it (which you know is correct, because Omega has already arbitrarily demonstrated it's power and trustworthiness) means you are dealing with an outcome pump attached directly to your utility function. Since the structure of the problem gives it a great deal of room in which to operate, the only real limitation is the nature of your own utility function. Sure, it's entirely possible that your utility function could be laid out in such a way as to strongly emphasize the disutility of misinformation... but that just limits the nice things Omega can do for you, it does nothing to save you from the bad things it can do to you. It remains valid to show you a picture and say 'the picture you are looking at is a basilisk; it causes any human that sees it to die within 48 hours'. Even without assuming basilisks, you're still dealing with a hostile outcome pump. There's bound to be some truth that you haven't considered that will lead you to a bad end. And if you want to examine it in terms of Everett branches, Omega is arbitrarily powerful. It has the power to compute all possible universes and give you the information which has maximally bad consequences for your utility function in aggregate across all possible universes (this implies, of course, that Omega is outside the Matrix, but pretty much any problem invoking Omega does that).
Even so, Omega doesn't assure you of anything regarding the specific weights of the two pieces of information. Utility functions differ, and since there's nothing Omega could say that would be valid for all utility functions, there's nothing it will say at all. It's left to you to decide which you'd prefer.
That said, I do find it interesting to note under which lines of reasoning people will choose something labelled 'maximum disutility'. I had thought it to be a more obvious problem than that.
"You are not perfectly rational" is certainly an understatement, and it does seem to be an excellent catch-all for ways in which a non-brain-melting truth might be dangerous to me... but by that token, a utility-improving falsehood might be quite dangerous to me too, no? It's unlikely that my current preferences can accurately be represented by a self-consistent utility function, and since my volition hasn't been professionally extrapolated yet, it's easy to imagine false utopias that might be an improvement by the metric of my current "utility function" but turn out to be dystopian upon actual experience.
Suppose someone's been brainwashed to the point that their utility function is "I want to obey The Leader as best as I can" - do you think that after reflection they'd be better off with a utility-maximizing falsehood or with a current-utility-minimizing truth?
The problem does not concern itself with merely 'better off', since a metric like 'better off' instead of 'utility' implies 'better off' as defined by someone else. Since Omega knows everything you know and don't know (by the definition of the problem, since it's presenting (dis)optimal information based on it's knowledge of your knowledge), it is in a position to extrapolate your utility function. Accordingly, it maximizes/minimizes for your current utility function, not its own, and certainly not some arbitrary utility function deemed to be optimal for humans by whomever. If your utility function is such that you hold the well-being of another above yourself (maybe you're a cultist of some kind, true... but maybe you're just a radically altruistic utilitarian), then the results of optimizing your utility will not necessarily leave you any better off. If you bind your utility function the aggregate utility of all humanity, then maximizing that is something good for all humanity. If you bind it to one specific non-you person, then that person gets a maximized utility. Omega does not discriminate between the cases... but if it is trying to minimize your long-term utility, a handy way to do so is to get you to act against your current utility function.
Accordingly, yes; a current-utility-minimizing truth could possibly be 'better' by most definitions for a cultist then a current-utility-maximizing falsehood. Beware, though; reversed stupidity is not intelligence. Being convinced to ruin Great Leader's life or even murder him outright might be better for you than blindly serving him and making him dictator of everything, but that hardly means there's nothing better you could be doing. The fact that there exists a class of perverse utility functions which have negative consequences for those adopting them (and which can thus be positively reversed) does not imply that it's a good idea to try inverting your utility function in general.
In response to
Problem of Optimal False Information
Would the following be a True Fact that is supported by evidence?
You open the white box, and are hit by a poison dart, which causes you to drop into a irreversible, excruciatingly painful, minimally aware, coma, where by all outward appearances you look fine, and you find out the world goes downhill, while you get made to live forever, while still having had enough evidence that Yes, the dart DID in fact contain a poison that drops you into an:
irreversible(Evidence supporting this, you never come out of a coma),
excruciatingly painful(Evidence supporting this, your nerves are still working inside your head, you can feel this excruciating pain)
minimally aware (Evidence supporting this, while you are in the coma you are still vaugely aware that you can confirm all of this and hear about bad news that makes you feel worse on a level in addition to physical pain, such as being given the old poison dart because someone thinks it's a treasured memento instead of a constant reminder that you are an idiot.)
coma(Evidence supporting this, you can't actually act upon the outer world as if you were conscious),
where by all outward appearances you look fine(Evidence supporting this, no one appears to be aware that you are in utter agony to the point where you would gladly accept a mercy kill.)
and you find out the world goes downhill (Evidence supporting this, while in a minimally aware state, you hear about the world going downhill, UFAI, brutal torture, nuclear bombs, whatever bad things you don’t want to hear about.)
while you get made to live forever: (Evidence supporting this, you never, ever die.)
I mean, the disutility would probably be worse than that, but... surely you never purposely pick a CERTAINTY of such an optimized maximum disutility, regardless of what random knowledge it might comes with. It would be one thing if the knowledge was such that it was going to be helpful, but since it comes as part and parcel of a optimized maximum disutility, the knowledge is quite likely to be something useless or worse, like “Yes, this dart really did contain a poison to hit you with optimized maximum disutility, and you are now quite sure that is true." (You would probably have been sure of that well before now even if it wasn't explicitly given to you as a true fact by Omega!)
And Omega didn't mislead you, the dart REALLY was going to be that bad in the class of facts about darts!
Since that (or worse) seems likely to be the White Box, I'll probably as carefully as possible select the Black box while trying to be extremely sure that I didn't accidentally have a brain fart and flip the colors of the boxes by mistake in sheer panic. Anyone who would pick the White box intentionally doesn't seem to be giving enough credence to just how bad Omega can make a certainty of optimized maximum disutility and how useless Omega can select the true fact to be.
As stated, the only trap the white box contains is information... which is quite enough, really. A prediction can be considered a true statement if it is a self-fulfilling prophecy, after all. More seriously, if such a thing as a basilisk is possible, the white box will contain a basilisk. Accordingly, it's feasible that the fact could be something like "Shortly after you finish reading this, you will drop into an irreversible, excruciatingly painful, minimally aware coma, where by all outward appearances you look fine, yet you find out the world goes downhill while you get made to live forever", and there's some kind of sneaky pattern encoded in the pattern of the text and the border of the page or whatever that causes your brain to lock up and start firing pain receptors, such that the pattern is self-sustaining. Everything else about the world and living forever and such would have to have been something that would have happened anyway, lacking your action to prevent it, but if Omega knows UFAI will happen near enough in the future, and knows that such a UFAI would catch you in your coma and stick you with immortality nanites without caring about your torture-coma state... then yeah, just such a statement is entirely possible.
In response to
Problem of Optimal False Information
The answer to this problem is only obvious because it's framed in terms of utility. Utility is, by definition, the thing you want. Strictly speaking, this should include any utility you get from your satisfaction at knowing the truth rather than a lie.
So for someone who valued knowing the truth highly enough, this problem actually should be impossible for Omega to construct.
Okay, so you are a mutant, and you inexplicably value nothing but truth. Fine.
The falsehood can still be a list of true things, tagged with 'everything on this list is true', but with an inconsequential falsehood mixed in, and it will still have net long-term utility for the truth-desiring utility function, particularly since you will soon be able to identify the falsehood, and with your mutant mind, quickly locate and eliminate the discrepancy.
The truth has been defined as something that cannot lower the accuracy of your beliefs, yet it still has maximum possible long-term disutility, and your utility function is defined exclusively in terms of the accuracy of your beliefs. Fine. Mutant that you are, the truth of maximum disutility is one which will lead you directly to a very interesting problem that will distract you for an extended period of time, but which you will ultimately be unable to solve. This wastes a great deal of your time, but leaves you with no greater utility than you had before, constituting disutility in terms of the opportunity cost of that time which you could've spent learning other things. Maximum disutility could mean that this is a problem that will occupy you for the rest of your life, stagnating your attempts to learn much of anything else.
In response to
Problem of Optimal False Information
By what sort of mechanism does a truth which will not be misleading in any way or cause me to lower the accuracy of any probability estimates nevertheless lead to a reduction in my utility? Is the external world unchanged, but my utility is lowered merely by knowing this brain-melting truth? Is the external world changed for the worse by differing actions of mine, and if so then why did I cause my actions to differ, given that my probability estimate for the false-and-I-already-disbelieved-it statement "these new actions will be more utility-optimal" did not become less accurate?
The problem is that truth and utility are not necessarily correlated. Knowing about a thing, and being able to more accurately assess reality because of it, may not lead you to the results you desire. Even if we ignore entirely the possibility of basilisks, which are not ruled out by the format of the question (eg: there exists an entity named Hastur, who goes to great lengths to torment all humans that know his name), there is also knowledge you/mankind are not ready for (plan for a free-energy device that works as advertised, but when distributed and reverse-engineered, leads to an extinction-causing physics disaster). Even if you yourself are not personally misled, you are dealing with an outcome pump that has taken your utility function into account. Among all possible universes, among all possible facts that fit the pattern, there has to be at least one truth that will have negative consequences for whatever you value, for you are not perfectly rational. The most benign possibilities are those that merely cause you to reevaluate your utility function, and act in ways that no longer maximize what you once valued; and among all possibilities, there could be knowledge which will do worse. You are not perfectly rational; you cannot perfectly foresee all outcomes. A being which has just proved to you that it is perfectly rational, and can perfectly foresee all outcomes, has advised you that the consequences of you knowing this information will be the maximum possible long-term disutility. By what grounds do you disbelieve it?
In response to
Problem of Optimal False Information
I would pick the black box, but it's a hard choice. Given all the usual suppositions about Omega as a sufficiently trustworthy superintelligence, I would assume that the utilities really were as it said and take the false information. But it would be a painful, both because I want to be the kind of person who pursues and acts upon the truth, and also because I would be desperately curious to know what sort of true and non-misleading belief could cause that much disutility -- was Lovecraft right after all? I'd probably try to bargain with Omega to let me know the true belief for just a minute before erasing it from my memory -- but still, in the Least Convenient Possible World where my curiosity was never satisfied, I'd hold my nose and pick the black box.
Having answered the hypothetical, I'll go on and say that I'm not sure there's much to take from it. Clearly, I don't value Truth for its own sake over and beyond all other considerations, let the heavens fall -- but I never thought I did, and I doubt many here do. The point is that in the real world, where we don't yet have trustworthy superintelligences, the general rule that your plans will go better when you use an accurate map doesn't seem to admit of exceptions (and little though I understand Friendly AI, I'd be willing to bet that this rule holds post-singularity). Yes, there are times where you might be better off with a false belief, but you can't predictably know in advance when that is, black swan blow-ups, etc.
To be more concrete, I don't think there's any real-world analogue to the hypothetical. If a consortium of the world's top psychiatrists announced that, no really, believing in God makes people happier, more productive, more successful, etc., and that this conclusion holds even for firm atheists who work for years to argue themselves into knots of self-deception, and that this conclusion has the strongest sort of experimental support that you could expect in this field, I'd probably just shrug and say "I defy the data". When it comes to purposeful self-deception, it really would take Omega to get me on board.
That's exactly why the problem invokes Omega, yes. You need an awful lot of information to know which false beliefs actually are superior to the truth (and which facts might be harmful), and by the time you have it, it's generally too late.
That said, the best real-world analogy that exists remains amnesia drugs. If you did have a traumatic experience, serious enough that you felt unable to cope with it, and you were experiencing PTSD or depression related to the trauma that impeded you from continuing with your life... but a magic pill could make it all go away, with no side effects, and with enough precision that you'd forget only the traumatic event... would you take the pill?
In response to
Problem of Optimal False Information
Can I inject myself with a poison that will kill me within a few minutes and THEN chose the falsehood?
Suicide is always an option. In fact, Omega already presented you with it as an option, the consequences for not choosing. If you would in general carry around such a poison with you, and inject it specifically in response to just such a problem, then Omega would already know about that, and the information it offers would take that into account. Omega is not going to give you the opportunity to go home and fetch your poison before choosing a box, though.
EDIT: That said, I find it puzzling that you'd feel the need to poison yourself before choosing the falsehood, which has already been demonstrated to have positive consequences for you. Personally, I find it far easier to visualize a truth so terrible that it leaves suicide the preferable option.
I'd say that this is too optimistic. Omega checks the future and if, in fact, you would eventually win the lottery if you started playing, then deluding you about lotteries might be a good strategy. But for most people that Omega talks to, this wouldn't work.
It's possible that the number of falsehoods that have one-in-a-million odds of helping you exceeds a million by far, and then it's very likely that Omega (being omniscient) can choose one that turns out to be helpful. But it's more interesting to see if there are falsehoods that have at least a reasonably large probability of helping you.
True; being deluded about lotteries is unlikely to have positive consequences normally, so unless something weird is going to go on in the future (eg: the lottery machine's random number function is going to predictably malfunction at some expected time, producing a predictable set of numbers; which Omega then imposes on your consciousness as being 'lucky'), that's not a belief with positive long-term consequences. That's not an impossible set of circumstances, but it is an easy-to-specify set, so in terms of discussing 'a false belief which would be long-term beneficial', it leaps readily to mind.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
That's the problem. The question is the rationalist equivalent of asking "Suppose God said he wanted you to kidnap children and torture them?" I'm telling Omega to just piss off.
The bearing this has on applied rationality is that this problem serves as a least convenient possible world for strict attachment to a model of epistemic rationality. Where the two conflict, you should probably prefer to do what is instrumentally rational over what is epistemically rational, because it's rational to win, not complain that you're being punished for making the "right" choice. As with Newcomb's Problem, if you can predict in advance that the choice you've labelled "right" has less utility than a "wrong" choice, that implies that you have made an error in assessing the relative utilities of the two choices. Sure, Omega's being a jerk. It does that. But that doesn't change the situation, which is that you are being presented with a situation where you are asked to choose between two situations of differing utility, and being trapped into an option of lesser utility (indeed, vastly lesser utility) by nothing but your own "rationality". This implies a flaw in your system of rationality.