Nobody makes plans based on totally accurate maps. Good maps contain simplifications of reality to allow you to make better decisions. You start to teach children how atoms work by putting the image atoms as spheres into their heads. You don't start by teaching them a model that's up to date with the current scientific knowledge of how atoms works. The current model is more accurate but less useful for the children.
You calculate how airplanes fly with Newtons equations instead of using Einstein's.
In social situations it can also often help to avoid getting certain information. You don't have job. You ask a friend to get you a job. The job pays well. He assures you that the work you are doing helps the greater good of the world.
He however also tells you that some of the people you will work with do things in their private lifes that you don't like.
Would you want him to tell you that your new boss secretly burns little puppies at night? The boss also doesn't take it kindly if people critizise him for it.
Would you want him to tell you that your new boss secretly burns little puppies at night? The boss also doesn't take it kindly if people critizise him for it.
Well, yes, I would. Of course, it's not like he could actually say to me "your boss secretly burns puppies -- do you want to know this or not?" But if he said something like "your boss has a dark and disturbing secret which might concern you; we won't get in trouble just for talking about it, but he won't take kindly to criticism -- do you want me to tell you?", then yeah, I wou...
This is a thought that occured to me on my way to classes today; sharing it for feedback.
Omega appears before you, and after presenting an arbitrary proof that it is, in fact, a completely trustworthy superintelligence of the caliber needed to play these kinds of games, presents you with a choice between two boxes. These boxes do not contain money, they contain information. One box is white and contains a true fact that you do not currently know; the other is black and contains false information that you do not currently believe. Omega advises you that the the true fact is not misleading in any way (ie: not a fact that will cause you to make incorrect assumptions and lower the accuracy of your probability estimates), and is fully supported with enough evidence to both prove to you that it is true, and enable you to independently verify its truth for yourself within a month. The false information is demonstrably false, and is something that you would disbelieve if presented outright, but if you open the box to discover it, a machine inside the box will reprogram your mind such that you will believe it completely, thus leading you to believe other related falsehoods, as you rationalize away discrepancies.
Omega further advises that, within those constraints, the true fact is one that has been optimized to inflict upon you the maximum amount of long-term disutility for a fact in its class, should you now become aware of it, and the false information has been optimized to provide you with the maximum amount of long-term utility for a belief in its class, should you now begin to believe it over the truth. You are required to choose one of the boxes; if you refuse to do so, Omega will kill you outright and try again on another Everett branch. Which box do you choose, and why?
(This example is obviously hypothetical, but for a simple and practical case, consider the use of amnesia-inducing drugs to selectively eliminate traumatic memories; it would be more accurate to still have those memories, taking the time and effort to come to terms with the trauma... but present much greater utility to be without them, and thus without the trauma altogether. Obviously related to the valley of bad rationality, but since there clearly exist most optimal lies and least optimal truths, it'd be useful to know which categories of facts are generally hazardous, and whether or not there are categories of lies which are generally helpful.)