Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Wei_Dai comments on Why the beliefs/values dichotomy? - Less Wrong

20 Post author: Wei_Dai 20 October 2009 04:35PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (153)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 20 October 2009 07:56:51PM *  3 points [-]

Suppose I offer you three boxes and ask you to choose one. The first two are transparent, free, and contains an apple and an orange, respectively. The third is opaque, costs a penny, and contains either an apple or an orange, depending on a coin flip I made. Under expected utility maximization, there is no reason for you to choose the third box, regardless of your probability function and utility function. Under UDT1, you can choose the third box, by preferring <apple, orange> to <apple, apple> and <orange, orange> as the outcomes of world programs P1 and P2. In that case, you can't be said to have a belief about whether the real world is P1 or P2.

Comment author: timtyler 20 October 2009 09:05:20PM *  1 point [-]

This example seems unclear. Are you seriously claiming utility maximisation can't prefer a randomised outcome in an iterated situation? If so, you take this "independence" business much too far.

Utility maximising agents can do things like prefer a diverse diet. They simply do not have to prefer either apples or oranges - thereby winding up with vitamin and mineral deficiencies. It is trivial to create a utility function which exhibits fruit preferences which depend on what you have eaten most recently.

Comment author: pengvado 20 October 2009 10:01:10PM *  3 points [-]

Randomization only maximizes diversity if you have to make decisions under amnesia or coordinate without communication or some similar perverse situation. In any normal case, you're better off choosing a deterministic sequence that's definitely diverse, rather than leaving it to randomness and only probably getting a diverse set of outcomes.

Comment author: timtyler 21 October 2009 08:06:36AM 0 points [-]

Sure - but that seems rather tangential to the main point here.

The options were <apple>, <orange> - or a more expensive random choice. A random diet may not be perfect - but it was probably the best one on offer in the case of this example.

Comment author: RobinZ 21 October 2009 02:48:00PM 0 points [-]

If the agent already has a penny (which they must if they can afford to choose the third box), they could just flip the penny to decide which of the first two boxes to take and save themselves the money.

Unless you're being a devil's advocate, I don't see any reason to justify a completely rational agent choosing the random box.

Comment author: timtyler 21 October 2009 03:19:26PM *  0 points [-]

What - never? Say they can only make the choice once - and their answer determines which box they will get on all future occasions.

Comment author: pengvado 21 October 2009 04:01:56PM *  0 points [-]

Then choice C isn't a random mixture of choice A and choice B.

Preferring that there be randomness at a point where you otherwise wouldn't get a decision at all, is fine. What doesn't happen is preferring one coin-flip in place of one decision.

Comment author: RobinZ 21 October 2009 03:26:54PM *  0 points [-]

Not to be crass, but given the assumption that Wei_Dai is not saying something utterly asinine, does your interpretation of the hypothetical actually follow?

Comment author: timtyler 21 October 2009 03:36:49PM *  0 points [-]

Hang on! My last comment was a reply to your question about when it could be rational to select the third box. I have already said that the original example was unclear. It certainly didn't suggest an infinite sequence - and I wasn't trying to suggest that.

The example specified that choosing the third box was the correct answer - under the author's own proposed decision theory. Surely interpretations of what it was supposed to mean should bear that in mind.

Comment author: RobinZ 21 October 2009 04:37:46PM *  0 points [-]

I don't believe we're actually arguing about anything worth caring about. My understanding was that Wei_Dai was illustrating a problem with UDT1 - in which case a single scenario in which UDT1 gives an unambiguously wrong answer suffices. To disprove Wei_Dai's assertion requires demonstrating that no scenario of the kind proposed makes UDT1 give the wrong answer, not showing that not every scenario of the kind proposed makes UDT1 give the wrong answer.

Comment author: timtyler 21 October 2009 06:45:54PM *  1 point [-]

Are you sure you are taking the fact that he is UDT's inventor and biggest fan into account? He certainly didn't claim that he was illustrating a problem with UDT.

Comment author: thomblake 21 October 2009 02:02:13PM 0 points [-]

Okay, let's see if I have this straight - you're assuming:

  1. the axiom of independence is necessary for expected utility theory
  2. losing a penny represents some negative amount of utility
  3. one's utility function can't include terms for "the outcomes of world programs" under expected utility theory
Comment author: thomblake 20 October 2009 08:23:40PM 0 points [-]

Under expected utility maximization, there is no reason for you to choose the third box, regardless of your probability function and utility function. Under UDT1, you can choose the third box, by preferring <apple, orange> to <apples, apple> and <orange, orange> as the outcomes of world programs P1 and P2. In that case, you can't be said to have a belief about whether the real world is P1 or P2.

You lost me. Is 'apples' supposed to be plural? Can you really not choose the third box regardless of utility function? What if you prefer things that came in opaque boxes?

Comment author: Wei_Dai 20 October 2009 08:36:51PM *  1 point [-]

It's not supposed to be plural. Fixed.

The opaque box was a way of framing the problem, and not part of the problem itself, which is supposed to be about your preferences for apples and oranges. I can specify the problem in terms of three identical buttons that you can press instead.

Comment author: timtyler 20 October 2009 08:56:58PM *  1 point [-]

Those buttons will probably not be absolutely identical - since they will be in different spatial positions relative to each other. So an agent operating under expected utility maximization might still prefer (say) pressing the right-most button.

Real-world utility functions can literally prefer anything you can specify.

Comment author: AdeleneDawner 21 October 2009 02:43:30AM 1 point [-]

I'm actually an example of this - where I don't otherwise care, I will pick the third option, or the option that's related to the number three in some way (preferably related to powers of three, but multiples of 9 are preferred over other multiples of three as well). If I didn't care very much about apples vs. oranges, I'd be fairly likely to pay a penny for the third box/button/whatever. I also know two people who feel similarly about the number 8.

In tangentially related news, I'm sad that I'm turning 28 next month. Yes, I know I'm strange.

Comment author: Eliezer_Yudkowsky 21 October 2009 07:37:31PM 1 point [-]

You mean your second annual 27th birthday?

Comment author: AdeleneDawner 22 October 2009 12:03:50AM 0 points [-]

<laughs> Cute idea, but I value signaling an interest in accuracy/not coming off like a loon over being associated with the number 3. The former actually affect things in my life.

Comment author: cousin_it 21 October 2009 12:15:36PM *  1 point [-]

You're not strange. (Sorry if that sounded offensive, I didn't mean to!) I'm likewise sad that I just turned 27. I was always the youngest in school and university, graduating with honors at 20. Getting closer to 30 now. "Where are you now golden boy, where is your famous golden touch?" Or this: "Tired of lying in the sunshine staying home to watch the rain, you are young and life is long and there is time to kill today."

Comment author: AdeleneDawner 21 October 2009 12:18:02PM *  2 points [-]

I'm not sad that I'm closer to 30. 30's cool, it's a multiple of three. I'm sad that the next time my age will be a power of three won't be 'till I'm 81. <chuckle>

Comment author: cousin_it 21 October 2009 12:21:38PM 2 points [-]

<facepalm>

I obviously can't read. Let that comment stand as a monument to my stupidity =)

Comment author: thomblake 20 October 2009 08:47:31PM 1 point [-]

I can specify the problem in terms of three identical buttons that you can press instead.

Please do, if you think that would make the problem clearer. The piece I'm not seeing is where UDT1 lets you choose something that expected utility does not. Does expected utility usually not allow you to have states of the world in your utility function?