gjm comments on How Not to be Stupid: Know What You Want, What You Really Really Want - Less Wrong

0 Post author: Psy-Kosh 28 April 2009 01:11AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

You are viewing a single comment's thread.

Comment author: gjm 28 April 2009 08:52:06AM 8 points [-]

I don't think you've shown very convincingly that it's always wrong to have two states that you're simply unable to compare with one another. The notion isn't inherently incoherent (as you can see, e.g., from the fact that there are mathematical structures that work that way, such as Conway-style combinatorial games) and it needn't lead to horrors like your two separate and incompatible series. In any case, your argument about those series is itself confused; if you know that you can do A1 -> B2 and then B2 -> A3, then you know that you can do A1 -> A3, and you will certainly do that. The fact that the transition happens via B2 is just an implementation detail, and there's no point pretending that you can't see past it. If you don't know about B2 -> A3 when you have to choose whether to do A1 -> B2, though, your problem is just ignorance, and there's nothing irrational about sometimes reaching suboptimal decisions when you don't have all the relevant information.

Actually, the confused-with relation between Conway games basically means "there are C,D such that A+C>B+C but A+D<B+D", which rather suggests that some structure along those lines might be appropriate for modelling preferences over incomplete states. Realistically, of course, all our preferences are over incomplete states; we have very limited information and very limited minds. Which is one reason why it seems excessive to me to make claims about our preferences that implicitly assume that we're working with complete states-of-the-universe all the time.

And, speaking of Conway games, the closely related Conway numbers ("surreal numbers", as they are usually called) show that even if your preferences are totally ordered it's not obvious that they can be embedded into the real numbers. Of course, if you take advantage of the finiteness of your brain to point out that you only have finitely many possible preferences then all is well -- but then you lose in a different way, because if you take that into account then you also have to bid farewell to all hope of a total ordering over all states.

The point of all this quibbling is simply this: if you are going to claim that a particular way of thinking is rationally mandatory then you need to either deal with all the little holes or acknowledge them. For my part, I don't think we yet have a firm enough theoretical foundation to claim that a rational agent's preferences (or, anticipating a bit, credences) must be representable by real numbers.

(Also, typo alert: there's a missing word somewhere after "the universe or an opponent or whatever could"; and you have "rankled" for "ranked" a few paragraphs after that.)

Comment author: AdeleneDawner 28 April 2009 09:28:49PM 1 point [-]

Replacing A with 'coffee' and B with 'tea' may be useful, here. It seems reasonable to me to not know offhand whether you prefer coffee or tea - I suspect most people have never thought about that directly - but most people would still know that they'd prefer (for example) an espresso from Starbucks (A1) to a cup of Earl Grey (B1), and either of those to a cup of coffee from the local diner where the coffee always tastes like soap (A2).

Comment author: Nick_Tarleton 28 April 2009 05:20:26PM 1 point [-]

And, speaking of Conway games, the closely related Conway numbers ("surreal numbers", as they are usually called) show that even if your preferences are totally ordered it's not obvious that they can be embedded into the real numbers.

If you have lexically ordered 'orders' of utility, only the highest order will ever affect your actions in non-toy situations, and you might as well use reals.

Comment author: gjm 28 April 2009 07:38:57PM 0 points [-]

I think that's debatable. For instance, consider Eliezer's "torture versus dust specks" question from way back on OB. (In case you weren't reading OB then or have forgotten: let N be a vastly unimaginably huge number (Eliezer chose 3^^^3 in Knuth arrow notation), and ask: which is worse, for one person to be tortured horribly for 50 years or for N people each to get a small speck of dust in their eye, just enough to be marginally annoying, but to suffer no longer-term consequences from it? I claim that having separate "orders" of utility is at most as irrational as choosing SPECKS rather than TORTURE, and that it's at least arguable that SPECKS is a defensible answer.

Comment author: Cyan 28 April 2009 07:55:45PM *  1 point [-]

I claim that having separate "orders" of utility is at most as irrational...

The point isn't about the (ir)rationality of separate "orders" of utility. It's a "without loss of generality" argument. Preferences not found at the highest order are effectively irrelevant, so you don't lose any expressive power by restricting yourself to the reals.

Comment author: gjm 28 April 2009 10:42:12PM *  0 points [-]

Er, sorry, I was unclear. (I wrote unclearly because I wasn't thinking clearly enough. It's annoying how that happens.) So, the point I was trying to make but didn't actually get around to writing down because I forgot about it while writing down what I did :-) is that those people for whom dust specks and torture are incommensurable -- which I think they have to be, to prefer 3^^^3.SPECK to 1.TORTURE -- don't, so far as I can tell, generally spend their entire lives estimating how many people are going to get tortured-or-worse on account of their actions, neither do they entirely ignore minor inconveniences; so it doesn't seem to be the case that having that sort of utility function implies ignoring everything but the highest order.

[EDITED above, about a day after posting, to fix a formatting glitch that I hadn't noticed before.]

Arguably it would do if those people were perfectly consistent -- one of the more convincing arguments for preferring TORTURE to SPECKS consists of exhibiting a series of steps between SPECK and TORTURE of length, say, at most 100 in which no step appears to involve a worse than, say, 100:1 difference in badness, so maybe preferring TORTURE to SPECKS almost always involves intransitivity or something like that. And maybe some similar charge could be brought against anyone who has separate "orders" but still gives any consideration to the lower ones. Hence my remark that the one doesn't seem more irrational than the other.

Comment author: Cyan 28 April 2009 03:22:22PM 0 points [-]

For my part, I don't think we yet have a firm enough theoretical foundation to claim that a rational agent's preferences (or, anticipating a bit, credences) must be representable by real numbers.

I think this is correct. On the other hand, I'm happy take real numbers as a point of departure just to see what we can get.