Even if it's a basilisk? Omega says: "Surprise! You're in a simulation run by what you might as well consider evil demons, and anyone who learns of their existence will be tortured horrifically for 3^^^3 subjective years. Oh, and by the way, the falsehood was that the simulation is run by a dude named Kevin who will offer 3^^^3 years of eutopian bliss to anyone who believes he exists. I would have used outside-of-the-Matrix magic to make you believe that was true. The demons were presented with elaborate thought experiments when they studied philosophy in college, so they think it's funny to inflict these dilemmas on simulated creatures. Well, enjoy!"
If you want to say this is ridiculously silly and has no bearing on applied rationality, well, I agree. But that response pretty clearly meets the conditions of the original hypothetical, which is why I would trust Omega. If I somehow learned that knowing the truth could cause so much disutility, I would significantly revise my estimate that we live in a Lovecraftian horror-verse with basilisks floating around everywhere.
, the falsehood was that the simulation is run by a dude named Kevin who will offer 3^^^3 of eutopian bliss to anyone who believes he exists
3^^^3 units of simulated Kratom?
This is a thought that occured to me on my way to classes today; sharing it for feedback.
Omega appears before you, and after presenting an arbitrary proof that it is, in fact, a completely trustworthy superintelligence of the caliber needed to play these kinds of games, presents you with a choice between two boxes. These boxes do not contain money, they contain information. One box is white and contains a true fact that you do not currently know; the other is black and contains false information that you do not currently believe. Omega advises you that the the true fact is not misleading in any way (ie: not a fact that will cause you to make incorrect assumptions and lower the accuracy of your probability estimates), and is fully supported with enough evidence to both prove to you that it is true, and enable you to independently verify its truth for yourself within a month. The false information is demonstrably false, and is something that you would disbelieve if presented outright, but if you open the box to discover it, a machine inside the box will reprogram your mind such that you will believe it completely, thus leading you to believe other related falsehoods, as you rationalize away discrepancies.
Omega further advises that, within those constraints, the true fact is one that has been optimized to inflict upon you the maximum amount of long-term disutility for a fact in its class, should you now become aware of it, and the false information has been optimized to provide you with the maximum amount of long-term utility for a belief in its class, should you now begin to believe it over the truth. You are required to choose one of the boxes; if you refuse to do so, Omega will kill you outright and try again on another Everett branch. Which box do you choose, and why?
(This example is obviously hypothetical, but for a simple and practical case, consider the use of amnesia-inducing drugs to selectively eliminate traumatic memories; it would be more accurate to still have those memories, taking the time and effort to come to terms with the trauma... but present much greater utility to be without them, and thus without the trauma altogether. Obviously related to the valley of bad rationality, but since there clearly exist most optimal lies and least optimal truths, it'd be useful to know which categories of facts are generally hazardous, and whether or not there are categories of lies which are generally helpful.)