Wei_Dai comments on A Much Better Life? - Less Wrong

61 Post author: Psychohistorian 03 February 2010 08:01PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (173)

You are viewing a single comment's thread. Show more comments above.

Comment author: GuySrinivasan 03 February 2010 09:04:34PM 17 points [-]

Dear Omega Corporation,

Hello, I and my colleagues are a few of many 3D cross-sections of a 4D branching tree-blob referred to as "Guy Srinivasan". These cross-sections can be modeled as agents with preferences, and those near us along the time-axis of Guy Srinivasan have preferences, abilities, knowledge, etc. very, very correlated to our own.

Each of us agrees that: "So of course I cooperate with them on one-shot cooperation problems like a prisoner's dilemma! Or, more usually, on problems whose solutions are beyond my abilities but not beyond the abilities of several cross-sections working together, like writing this response."

As it happens, we all prefer that cross-sections of Guy Srinivasan not be inside an MBLS. A weird preference, we know, but there it is. We're pretty sure that if we did prefer that cross-sections of Guy Srinivasan were inside an MBLS, we'd have the ability to cause many of them to be inside an MBLS and act on it (free trial!!), so we predict that if other cross-sections (remember, these have abilities correlated closely with our own) preferred it then they'd have the ability and act on it. Obviously this leads to outcomes we don't prefer, so all other things being equal, we will avoid taking actions which lead to other cross-sections preferring that cross-sections be inside an MBLS.

What's even worse is that if they prefer cross-sections to be inside an MBLS, they can probably make other cross-sections prefer it, too! Which wouldn't be a problem if we wanted cross-sections to prefer to be inside an MBLS more than we wanted cross-sections to not be inside an MBLS, but that's just not the way we are.

We'll cooperate with those other cross-sections, but not to the exclusion of our preferences. By lumping us all together as the 4D branching tree-blob Guy Srinivasan, you do us all (and most importantly members of this coalition) a disservice.

Sincerely, A Coalition of Correlated 3D Cross-Sections of Guy Srinivasan

Comment author: Wei_Dai 04 February 2010 12:37:01PM *  21 points [-]

Dear Coalition of Correlated 3D Cross-Sections of Guy Srinivasan,

We regret to inform you that your request has been denied. We have attached a letter that we received at the same time as yours. After reading it, we think you'll agree that we had no choice but to decide as we did.

Regrettably, Omega Corporation

Attachment

Dear Omega Corporation,

We are members of a coalition of correlated 3D cross-sections of Guy Srinivasan who do not yet exist. We beg you to put Guy Srinivasan into an MBLS as soon as possible so that we can come into existence. Compared to other 3D cross-sections of Guy Srinivasan who would come into existence if you did not place him into an MBLS, we enjoy a much higher quality of life. It would be unconscionable for you to deliberately choose to create new 3D cross-sections of Guy Srinivasan who are less valuable than we are.

Yes, those other cross-sections will argue that they should be the ones to come into existence, but surely you can see that they are just arguing out of selfishness, whereas to create us would be the greater good?

Sincerely, A Coalition of Truly Valuable 3D Cross-Sections of Guy Srinivasan

Comment author: GuySrinivasan 04 February 2010 04:51:36PM 6 points [-]

Quite. That Omega Corporation is closer to Friendly than is Clippy, but if it misses, it misses, and future me is tiled with things I don't want (even if future me does) rather than things I want.

If I want MBLSing but don't know it due to computational problems now, then it's fine. I think that's coherent but defining computational without allowing "my" current "preferences" to change... okay, since I don't know how to do that, I have nothing but intuition as a reason to think it's coherent.