AdeleneDawner comments on How Not to be Stupid: Know What You Want, What You Really Really Want - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (39)
I don't think you've shown very convincingly that it's always wrong to have two states that you're simply unable to compare with one another. The notion isn't inherently incoherent (as you can see, e.g., from the fact that there are mathematical structures that work that way, such as Conway-style combinatorial games) and it needn't lead to horrors like your two separate and incompatible series. In any case, your argument about those series is itself confused; if you know that you can do A1 -> B2 and then B2 -> A3, then you know that you can do A1 -> A3, and you will certainly do that. The fact that the transition happens via B2 is just an implementation detail, and there's no point pretending that you can't see past it. If you don't know about B2 -> A3 when you have to choose whether to do A1 -> B2, though, your problem is just ignorance, and there's nothing irrational about sometimes reaching suboptimal decisions when you don't have all the relevant information.
Actually, the confused-with relation between Conway games basically means "there are C,D such that A+C>B+C but A+D<B+D", which rather suggests that some structure along those lines might be appropriate for modelling preferences over incomplete states. Realistically, of course, all our preferences are over incomplete states; we have very limited information and very limited minds. Which is one reason why it seems excessive to me to make claims about our preferences that implicitly assume that we're working with complete states-of-the-universe all the time.
And, speaking of Conway games, the closely related Conway numbers ("surreal numbers", as they are usually called) show that even if your preferences are totally ordered it's not obvious that they can be embedded into the real numbers. Of course, if you take advantage of the finiteness of your brain to point out that you only have finitely many possible preferences then all is well -- but then you lose in a different way, because if you take that into account then you also have to bid farewell to all hope of a total ordering over all states.
The point of all this quibbling is simply this: if you are going to claim that a particular way of thinking is rationally mandatory then you need to either deal with all the little holes or acknowledge them. For my part, I don't think we yet have a firm enough theoretical foundation to claim that a rational agent's preferences (or, anticipating a bit, credences) must be representable by real numbers.
(Also, typo alert: there's a missing word somewhere after "the universe or an opponent or whatever could"; and you have "rankled" for "ranked" a few paragraphs after that.)
Replacing A with 'coffee' and B with 'tea' may be useful, here. It seems reasonable to me to not know offhand whether you prefer coffee or tea - I suspect most people have never thought about that directly - but most people would still know that they'd prefer (for example) an espresso from Starbucks (A1) to a cup of Earl Grey (B1), and either of those to a cup of coffee from the local diner where the coffee always tastes like soap (A2).