= 27d567bc2d48d9f40e9ccc20ea370b15)
I am worried about "a belief/fact in its class" the class chosen could have an extreme effect on the outcome.
In response to
Problem of Optimal False Information
As presented, the 'class' involved is 'the class of facts which fits the stated criteria'. So, the only true facts which Omega is entitled to present to you are those which are demonstrably true, which are not misleading as specified, which Omega can find evidence to prove to you, and which you could verify yourself with a month's work. The only falsehoods Omega can inflict upon you are those which are demonstrably false (a simple test would show they are false), which you do not currently believe, and which you would disbelieve if presented openly.
Those are fairly weak classes, so Omega has a lot of room to work with.
Except after executing the code, you'd know it was FAI and not a video game, which goes against the OP's rule that you honestly believe in the falsehood continually.
I guess it works if you replace "FAI" in your example with "FAI who masquerades as a really cool video game to you and everyone you will one day contact" or something similar, though.
The original problem didn't specify how long you'd continue to believe the falsehood. You do, in fact, believe it, so stopping believing it would be at least as hard as changing your mind in ordinary circumstances (not easy, nor impossible). The code for FAI probably doesn't run on your home computer, so there's that... you go off looking for someone who can help you with your video game code, someone else figures out what it is you're come across and gets the hardware to implement, and suddenly the world gets taken over. Depending on how attentive you were to the process, you might not correlate the two immediately, but if you were there when the people were running things, then that's pretty good evidence that something more serious then a video game happened.
In response to
Problem of Optimal False Information
The Bohr model of atomic structure is a falsehood which would have been of tremendous utility to a natural philosopher living a few hundred years ago.
That said, I feel like I'm fighting the hypothetical with that answer - the real question is, should we be willing to self-modify to make our map less accurate in exchange for utility? I don't think there's actually a clean decision-theoretic answer for this, that's what makes it compelling.
That is the real question, yes. That kind of self-modification is already cropping up, in certain fringe cases as mentioned; it will get more prevalent over time. You need a lot of information and resources in order to be able to generally self-modify like that, but once you can... should you? It's similar to the idea of wireheading, but deeper... instead of generalized pleasure, it can be 'whatever you want'... provided that there's anything you want more than truth.
Would the following be a valid falsehood? "The following program is a really cool video game: <code that is actually for a Friendly AI>"
I think we have a good contender for the optimal false information here.
The problem specifies that something will be revealed to you, which will program you to believe it, even though false. It doesn't explicitly limit what can be injected into the information stream. So yes, assuming you would value the existence of a Friendly AI, yes, that's entirely valid as optimal false information. Cost: you are temporarily wrong about something, and realize your error soon enough.
In response to
Problem of Optimal False Information
Does the utility calculation from the false belief include utility from the other beliefs I will have to overwrite? For example, suppose the false belief is "I can fly". At some point, clearly, I will have to rationalise away the pain of my broken legs from jumping off a cliff. Short of reprogramming my mind to really not feel the pain anymore - and then we're basically talking about wireheading - it seems hard to come up with any fact, true or false, that will provide enough utility to overcome that sort of thing.
I additionally note that the maximum disutility has no lower bound, in the problem as given; for all I know it's the equivalent of three cents. Likewise the maximum utility has no lower bound, in the problem as written. Perhaps Omega ought to provide some comparisons; for example he might say that the disutility of knowing the true fact is at least equal to breaking an arm, or some such calibration.
As the problem is written, I'd take the true fact.
As written, the utility calculation explicitly specifies 'long-term' utility; it is not a narrow calculation. This is Omega we're dealing with, it's entirely possible that it mapped your utility function from scanning your brain, and checked all possible universes forward in time from the addition of all possible facts to your mind, and took the worst and best true/false combination.
Accordingly, a false belief that will lead you to your death or maiming is almost certainly non-optimal. No, this is the one false thing that has the best long-term consequences for you, as you value such things, out of all the false things you could possibly believe.
True, the maximum utility/disutility has no lower bound. This is intentional. If you really believe that your position is such that no true information can hurt you, and/or no false information can benefit you, then you could take the truth. This is explicitly the truth with the worst possible long-term consequences for whatever it is you value.
Yes, it's pretty much defined as a sucker bet, implying that Omega is attempting to punish people for believing that there is no harmful true information and no advantageous false information. If you did, in fact, believe that you couldn't possibly gain by believing a falsehood, or suffer from learning a truth, this is the least convenient possible world.
In response to
Problem of Optimal False Information
You should choose the false belief, because Omega has optimized it for instrumental utility whereas the true belief has been optimized for disutility, and you may be vulnerable to such effects if only because you're not a perfectly rational agent.
If you were sure that no hazardous true information (or advantageous false information) could possibly exist, you should still be indifferent between the two choices: either of these would yield a neutral belief, leaving you with very nearly the same utility as before.
That is my point entirely, yes. This is a conflict between epistemic and instrumental rationality; if you value anything higher than truth, you will get more of it by choosing the falsehood. That's how the problem is defined.
In response to
Problem of Optimal False Information
Least optimal truths are probably really scary and to be avoided at all costs. At the risk of helping everyone here generalize from fictional evidence, I will point out the similarity to the Cthaeh in The Wise Man's Fear.
On the other hand, a reasonably okay falsehood to end up believing is something like "35682114754753135567 is prime", which I don't expect to affect my life at all if I suddenly start believing it. The optimal falsehood can't possibly be worse than that. Furthermore, if you value not being deceived about important things then the optimality of the optimal falsehood should take that into account, making it more likely that the falsehood won't be about anything important.
Edit: Would the following be a valid falsehood? "The following program is a really cool video game: <code that is actually for a Friendly AI>"
Yes, least optimal truths are really terrible, and the analogy is apt. You are not a perfect rationalist. You cannot perfectly simulate even one future, much less infinite possible ones. The truth can hurt you, or possibly kill you, and you have just been warned about it. This problem is a demonstration of that fact.
That said, if your terminal value is not truth, a most optimal falsehood (not merely a reasonably okay one) would be a really good thing. Since you are (again) not a perfect rationalist, there's bound to be something that you could be falsely believing that would lead you to better consequences than your current beliefs.
In response to
Problem of Optimal False Information
How one responds to this dilemma depends on how one values truth. I get the impression that while you value belief in truth, you can imagine that the maximum amount of long-term utility for belief in a falsehood is greater than the minimum amount of long-term utility for belief in a true fact. I would not be surprised to see that many others here feel the same way. After all, there's nothing inherently wrong with thinking this is so.
However, my value system is such that the value of knowing the truth greatly outweighs any* possible gains you might have from honestly believing a falsehood. I would reject being hooked up to Nozick's experience machine on utilitarian grounds: I honestly value the disutility of believing in a falsehood to be that bad.
(I am wary of putting the word "any" in the above paragraph, as maybe I'm not correctly valuing very large numbers of utilons. I'm not really sure how to evaluate differences in utility when it comes to things I really value, like belief in true facts. The value is so high in these cases that it's hard to see how anything could possibly exceed it, but maybe this is just because I have no understanding of how to properly value high value things.)
Okay, so if your utilities are configured that way, the false belief might be a belief you will encounter, struggle with, and get over in a few years, and be stronger for the experience.
For that matter, the truth might be 'your world is, in fact, a simulation of your own design, to which you have (through carelessness) forgotten the control codes; you are thus trapped and will die here, accomplishing nothing in the real world'. Obviously an extreme example; but if it is true, you probably do not want to know it.
In response to
Problem of Optimal False Information
First, your invocation of Everett branches adds nothing to the problem, as every instance of "you" may well decide not to choose. So, "choose or die" ought to be good enough, provided that you have a fairly strong dislike of dying.
Second, the traumatic memories example is great, but a few more examples would be useful. For example, the "truth" might be "discover LW, undergo religious deconversion, be ostracized by your family, get run over by a car while wondering around in a distraught state" whereas the "lie" might be "get hypnotized into believing Scientology, join the church, meet and hook up with a celebrity you dreamed about, live happily ever after".
Even more drastic: a bomb vs a wireheading device. The "truth" results in you being maimed, the "lie" results in you being happy, if out of touch with the world around you.
At this point it should be clear that there is no single "right" solution to the problem, and even the same person would choose differently depending on the situation, so the problem is not well defined as stated.
I didn't have any other good examples on tap when I originally conceived of the idea, but come to think of it...
Truth: A scientific formula, seemingly trivial at first, but whose consequences, when investigated, lead to some terrible disaster, like the sun going nova. Oops.
Lies involving 'good' consequences are heavily dependent upon your utility function. If you define utility in such a way that allows your cult membership to be net-positive, then sure, you might get a happily-ever-after cult future. Whether or not this indicates a flaw in your utility function is a matter of personal choice; rationality cannot tell you what to protect.
That said, we are dealing with Omega, who is serious about those optimals. This really is a falsehood with optimal net long-term utility for you. It might be something like a false belief about lottery odds, which leads to you spending the next couple years wasting large sums of money on lottery tickets... only to win a huge jackpot, hundreds of millions of dollars, and retire young, able to donate huge sums to the charities you consider important. You don't know, but it is, by definition, the best thing that could possibly happen to you as the result of believing a lie, as you define 'best thing'.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Hm. If there is a strong causal relationship between knowing truths and utility, then it is conceivable that this is a trick: the truth, while optimized for disutility, might still present me with a net gain over the falsehood and the utility. But honestly, I am not sure I buy that: you can get utility from a false belief, if that belief happens to steer you in such a way that it adds utility. You can't normally count on that, but this is Omega we are talking about.
The 'other related falsehoods and rationalizing' part has me worried. The falsehood might net me utility in and of itself, but I poke and prod ideas. If my brain becomes set up so that I will rationalize this thought, altering my other thought-patterns to match as I investigate... that could be very bad for me in the long run. Really bad.
And I've picked inconvenient truth over beneficial falsehood before on those grounds, so I probably would pick the inconvenient truth. Maybe, if Omega guaranteed that the process of rationalizing the falsehood would still have me with more utility- over the entire span of my life- than I would earn normally after you factor in the disutility of the inconvenient truth then I would take the false-box. But on the problem as it stands, I'd take the truth-box.
That's why the problem specified 'long-term' utility. Omega is essentially saying 'I have here a lie that will improve your life as much as any lie possibly can, and a truth that will ruin your life as badly as any truth can; which would you prefer to believe?'
Yes, believing a lie does imply that your map has gotten worse, and rationalizing your belief in the lie (which we're all prone to do to things we believe) will make it worse. Omega has specified that this lie has optimal utility among all lies that you, personally, might believe; being Omega, it is as correct in saying this as it is possible to be.
On the other hand, the box containing the least optimal truth is a very scary box. Presume first that you are particularly strong emotionally and psychologically; there is no fact that will directly drive you to suicide. Even so, there are probably facts out there that will, if comprehended and internalized, corrupt your utility function, leading you to work directly against all you currently believe in. There's probably something even worse than that out there in the space of all possible facts, but the test is rated to your utility function when Omega first encountered you, so 'you change your ethical beliefs, and proceed to spend your life working to spread disutility, as you formerly defined it' is on the list of possibilities.