Regardless of whether the third one is plausible, I suspect Omega would know of some hack that is equally weird and unable to be anticipated.

As stated, the question comes down to acting on an opinion you have on an unknown, but within the principles of this problem potentially knowable conclusion about your own utility function. And that is: Which is larger: 1) the amount of positive utility you gain from knowing the most disutile truths that exist for you OR 2) the amount of utility you gain from knowing the most utile falsehoods that exist for you

ALMOST by definition of the word utility, you would choose the truth (white box) if and only if 1) is larger and you would choose the falsehood (black box) only if 2) is larger. I say almost by definition because all answers of the form "I would choose the truth even if it was worse for me" are really statements that the utility you place on the truth is higher than Omega has assumed, which violates the assumption that Omega knows your utility function and speaks truthfully about it.

I say ALMOST by definition because we have to consider the other piece of the puzzle: when I open box 2) there is a machine that "will reprogram your mind." Does this change anything? Well it depends on which utility function Omega is using to make her calculations of my long term utility. Is Omega using my utility function BEFORE the machine reprogram's my mind, or after? Is me after the reprogramming really still me after the reprogramming? I think within the spirit of the problem we must assume that 1) The utility happens to be maximized for both me before the reprogram and me after the reprogram, perhaps my utility function does not change at all in the reprogramming, 2) Omega has correctly included the amount of disutility I would have to the particular programming change, and this is factored in to her calculations so that the proposed falsehood and mind reprogramming do in fact, on net, give the maximum utility I can get from knowing the falsehood PLUS being reprogrammed.

Within these constraints, we find that the "ALMOST" above can be removed if we include the (dis)utility I have for the reprogramming in the calculation. So:

Which is larger: 1) the amount of positive utility you gain from knowing the most disutile truths that exist for you AND being reprogrammed to believe it OR 2) the amount of utility you gain from knowing the most utile falsehoods that exist for you

So ultimately, the question which would we choose is the question above. I think to say anything else is to say "my utility is not my utility," i.e. to contradict yourself.

In my case, I would choose the white box. On reflection, considering the long run, I doubt that there is a falsehood PLUS a reprogramming that I would accept as a combination as more utile than the worst true fact (with no reprogramming to consider) that I would ever expect to get. Certainly, this is the Occam's razor answer, the ceteris paribus answer. GENERALLY, we believe that knowing more is better for us than being wrong. Generally we believe that someone else meddling with our minds has a high disutility to us.

For completeness I think these are straightforward conclusions from "playing fair" in this question, from accepting an Omega as postulated.

1) If Omega assures you the utility for 2) (including the disutility of the reprogramming as experienced by your pre-reprogrammed self) is 1% higher than the utility of 1), then you want to choose 2), to choose the falsehood and the reprogramming. To give any other answer is to presume that Omega is wrong about your utility , which violates the assumptions of the question.

2) If Omega assures you the utility for 2) and 1) are equal, it doesn't matter which one you choose. As much as you might think "all other things being equal I'll choose the truth" you must accept that the value you place on the truth has already been factored in, and the blip-up from choosing the truth will be balanced by some other disutility in a non-truth area. Since you can be pretty sure that the utility you place on the truth is very much unrelated to pain and pleasure and joy and love and so on, you are virtually guaranteeing you will FEEL worse choosing the truth, but that this worse feeling will just barely be almost worth it.

Finally, I tried to play nice within the question. But it is entirely possible, and I would say likely, that there can never be an Omega who could know ahead of time with the kind of detail required, what your future utility would be, at least not in our Universe. Consider just the quantum uncertainties (or future Everett universe splits). It seems most likely that your future net utility covers a broad range of outcomes in different Everett branches. In that case, it seems very likely that there is no one truth that minimizes your utility in all your possible futures, and no one falsehood that maximizes it in all your possible futures. In this case we would have a distribution of utility outcomes from 1) and 2) and it is not clear that we know how to choose between two different distributions. Possibly utility is defined in such a way that it would be the expectation value that "truly" mattered to us, but that puts I think a very serious constraint on utility functions and how we interact with them that I am not sure could be supported.

"You are not perfectly rational" is certainly an understatement, and it does seem to be an excellent catch-all for ways in which a non-brain-melting truth might be dangerous to me... but by that token, a utility-improving falsehood might be quite dangerous to me too, no? It's unlikely that my current preferences can accurately be represented by a self-consistent utility function, and since my volition hasn't been professionally extrapolated yet, it's easy to imagine false utopias that might be an improvement by the metric of my current "utility function" but turn out to be dystopian upon actual experience.

Suppose someone's been brainwashed to the point that their utility function is "I want to obey The Leader as best as I can" - do you think that after reflection they'd be better off with a utility-maximizing falsehood or with a current-utility-minimizing truth?

Would the following be a True Fact that is supported by evidence?

You open the white box, and are hit by a poison dart, which causes you to drop into a irreversible, excruciatingly painful, minimally aware, coma, where by all outward appearances you look fine, and you find out the world goes downhill, while you get made to live forever, while still having had enough evidence that Yes, the dart DID in fact contain a poison that drops you into an:

irreversible(Evidence supporting this, you never come out of a coma),

excruciatingly painful(Evidence supporting this, your nerves are still working inside your head, you can feel this excruciating pain)

minimally aware (Evidence supporting this, while you are in the coma you are still vaugely aware that you can confirm all of this and hear about bad news that makes you feel worse on a level in addition to physical pain, such as being given the old poison dart because someone thinks it's a treasured memento instead of a constant reminder that you are an idiot.)

coma(Evidence supporting this, you can't actually act upon the outer world as if you were conscious),

where by all outward appearances you look fine(Evidence supporting this, no one appears to be aware that you are in utter agony to the point where you would gladly accept a mercy kill.)

and you find out the world goes downhill (Evidence supporting this, while in a minimally aware state, you hear about the world going downhill, UFAI, brutal torture, nuclear bombs, whatever bad things you don’t want to hear about.)

while you get made to live forever: (Evidence supporting this, you never, ever die.)

I mean, the disutility would probably be worse than that, but... surely you never purposely pick a CERTAINTY of such an optimized maximum disutility, regardless of what random knowledge it might comes with. It would be one thing if the knowledge was such that it was going to be helpful, but since it comes as part and parcel of a optimized maximum disutility, the knowledge is quite likely to be something useless or worse, like “Yes, this dart really did contain a poison to hit you with optimized maximum disutility, and you are now quite sure that is true." (You would probably have been sure of that well before now even if it wasn't explicitly given to you as a true fact by Omega!)

And Omega didn't mislead you, the dart REALLY was going to be that bad in the class of facts about darts!

Since that (or worse) seems likely to be the White Box, I'll probably as carefully as possible select the Black box while trying to be extremely sure that I didn't accidentally have a brain fart and flip the colors of the boxes by mistake in sheer panic. Anyone who would pick the White box intentionally doesn't seem to be giving enough credence to just how bad Omega can make a certainty of optimized maximum disutility and how useless Omega can select the true fact to be.

The answer to this problem is only obvious because it's framed in terms of utility. Utility is, by definition, the thing you want. Strictly speaking, this should include any utility you get from your satisfaction at knowing the truth rather than a lie.

So for someone who valued knowing the truth highly enough, this problem actually should be impossible for Omega to construct.

By what sort of mechanism does a truth which will not be misleading in any way or cause me to lower the accuracy of any probability estimates nevertheless lead to a reduction in my utility? Is the external world unchanged, but my utility is lowered merely by knowing this brain-melting truth? Is the external world changed for the worse by differing actions of mine, and if so then why did I cause my actions to differ, given that my probability estimate for the false-and-I-already-disbelieved-it statement "these new actions will be more utility-optimal" did not become less accurate?

I would pick the black box, but it's a hard choice. Given all the usual suppositions about Omega as a sufficiently trustworthy superintelligence, I would assume that the utilities really were as it said and take the false information. But it would be a painful, both because I want to be the kind of person who pursues and acts upon the truth, and also because I would be desperately curious to know what sort of true and non-misleading belief could cause that much disutility -- was Lovecraft right after all? I'd probably try to bargain with Omega to let me know the true belief for just a minute before erasing it from my memory -- but still, in the Least Convenient Possible World where my curiosity was never satisfied, I'd hold my nose and pick the black box.

Having answered the hypothetical, I'll go on and say that I'm not sure there's much to take from it. Clearly, I don't value Truth for its own sake over and beyond all other considerations, let the heavens fall -- but I never thought I did, and I doubt many here do. The point is that in the real world, where we don't yet have trustworthy superintelligences, the general rule that your plans will go better when you use an accurate map doesn't seem to admit of exceptions (and little though I understand Friendly AI, I'd be willing to bet that this rule holds post-singularity). Yes, there are times where you might be better off with a false belief, but you can't predictably know in advance when that is, black swan blow-ups, etc.

To be more concrete, I don't think there's any real-world analogue to the hypothetical. If a consortium of the world's top psychiatrists announced that, no really, believing in God makes people happier, more productive, more successful, etc., and that this conclusion holds even for firm atheists who work for years to argue themselves into knots of self-deception, and that this conclusion has the strongest sort of experimental support that you could expect in this field, I'd probably just shrug and say "I defy the data". When it comes to purposeful self-deception, it really would take Omega to get me on board.

Can I inject myself with a poison that will kill me within a few minutes and THEN chose the falsehood?

I'd say that this is too optimistic. Omega checks the future and if, in fact, you would eventually win the lottery if you started playing, then deluding you about lotteries might be a good strategy. But for most people that Omega talks to, this wouldn't work.

It's possible that the number of falsehoods that have one-in-a-million odds of helping you exceeds a million by far, and then it's very likely that Omega (being omniscient) can choose one that turns out to be helpful. But it's more interesting to see if there are falsehoods that have at least a reasonably large probability of helping you.

Wow. This is particularly interesting to me, because I already felt this way without knowing why, not having consciously examined the feeling. I know that I already felt uncomfortable around gift-giving holidays, and this provides context to that; I don't particularly enjoy receiving incorrect things, and indeed, I have several boxes full of incorrect things following me around that I can't get rid of (even if I can't think of any reason to have or use a thing, it feels like losing hit points to dispose of it). For the same reason, I feel uncomfortable giving incorrect things, but the conventions of the relevant holidays makes it unacceptable to go looking for the kind of information I need to give correct things (having noticed this for a while, I switched over to exclusively giving gift certificates a couple years back, it seems to help).

Now that I've consciously identified the feeling, however, I can more proactively approach the problem, by communicating these feelings to those with whom I'd normally exchange gifts.