Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

# Mestroyer comments on Rationality Quotes September 2012 - Less Wrong

7 03 September 2012 05:18AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

Comment author: 05 September 2012 09:08:33PM 8 points [-]

"I wish for the result of the hypothetical nth wish I would make if I was allowed to make n wishes in the limit as n went to infinity each time believing that the next wish would be my only one and all previous wishes would be reversed, or if that limit does not exist, pick n = busy beaver function of Graham's number."

Comment author: 06 September 2012 07:19:34AM 1 point [-]

I can see a genie taking a shortcut here.

"In any story worth the tellin', that knows about the way of the world, the third wish is the one that undoes the harm the first two wishes caused."

— Granny Weatherwax, A Hat Full of Sky.

In short, the genie may well conclude that every m'th wish, for some m (Granny Weatherwax suggests here that 'm' is three) your wish would be to have never met the genie in the first place. At this point, if you're lucky, the genie will use a value of n that's a multiple of m. If you're unlucky, the genie will use a value of n that's km-1 for some integer k...

Alternatively, you'll end up with a genie who can't handle the math and does not understand what you're asking for.

Comment author: 06 September 2012 12:26:01AM 0 points [-]

I'm pretty sure this would result in the genie killing you.

Comment author: 06 September 2012 01:16:37AM 2 points [-]

Because I would wish to kill myself eventually? It's hard to imagine that I would do that, faced with unlimited wishes. If I got bored, I could just wish the boredom away.

Though on reflection this wish needs a safeguard against infinite recursion, and a bliss-universe for any simulated copies of me the genie creates to determine what my wishes would be.

Comment author: 06 September 2012 02:36:38AM 3 points [-]

Because I would wish to kill myself eventually?

In a sufficiently bad situation, you may wish for the genie to kill you because you think that's your only wish. It's not likely for any given wish, but would happen eventually (and ends the recursion, so that's one of the few stable wishes).

Comment author: 06 September 2012 06:31:14AM 1 point [-]

If I kill myself, there is no nth wish as n -> infinity, or a busy beaver function of Graham's numberth wish, so the first wish is wishing for something undefined.

Also, the probability of any of the individually improbable events where I kill myself happening is bounded above by the some of their probabilities, and they could be a convergent infinite series, if the probability of wanting to kill myself went down each time. Even though I stipulated that it's if I believed each wish was the last, I might do something like "except don't grant this wish if it would result in me wanting to kill myself or dying before I could consider the question" in each hypothetical wish. Or grant myself superintelligence as part of one of the hypothetical wishes, and come up with an even better safeguard when I found myself (to my great irrational surprise) getting another wish.

There is not even necessarily a tiny chance of wanting to kill myself. Good epistemology says to think there is, just in case, but some things are actually impossible. Using wishes to make it impossible for me to want to kill myself might come faster than killing myself.

Comment author: 06 September 2012 06:37:11AM 2 points [-]

If I kill myself, there is no nth wish as n -> infinity, or a busy beaver function of Graham's numberth wish, so the first wish is wishing for something undefined.

I think you're right, though I'm not sure that's exactly a good thing.

and they could be a convergent infinite series, if the probability of wanting to kill myself went down each time.

I see no particular reason to expect that to be the case.

Using wishes to make it impossible for me to want to kill myself might come faster than killing myself.

Excellent point. That might just work (though I'm sure there are still a thousand ways it could go mind-bogglingly wrong).

Comment author: 26 September 2012 06:13:02PM 0 points [-]

If you did eventually wish for death, then you would have judged that death is the best thing you can wish for, after having tried as many alternatives as possible.

Are you going to try to use your prior (I don't want to die) to argue with your future self who has experienced the results (and determines that you would be happier dying right now than getting any wish)?

Comment author: 26 September 2012 06:59:32PM 1 point [-]

I would not want to kill myself if my distant-future self wanted to die or wanted me to die immediately. I think it is much more likely that I would accidentally self-modify in a manner I wouldn't like if I reflected on it now and that would lead to wishing for death than that my current self with additional knowledge would chose death over omnipotence.

I don't think the factual question "would I be happier dying right now" would necessarily be the one that decided the policy question of "will I chose do die" for both me and my future self, because we could each care about different things.

And with a warning "The way things are going, you'll end up wanting to die." I could change my wishes and maybe get something better.

Comment author: 27 September 2012 05:43:15PM 0 points [-]

"Does my current utility function contain a global maximum at the case where I wish for and receive death right now?" is a really scary question to ask a genie.

I would prefer "I wish for the world to changed in such a manner as my present actual utility function, (explicitly distinct from my perception of my utility function), is at the global maximum possible without altering my present actual utility function."

Or in colloquial language "Give me what I want, not what I think I want."

Comment author: 27 September 2012 08:55:22PM 1 point [-]

That sounds pretty scary too. I don't think I am close enough to being an agent to have a well-defined utility function. If I do (paradoxical as it sounds), it's probably not something I would reflectively like. For example, I think I have more empathy for things I am sexually attracted to. But the idea of a world where everyone else (excluding me and a few people I really like) is a second-class citizen to hot babes horrifies me. But with the wrong kind of extrapolation, I bet that could be said to be what I want.

I can't easily describe any procedure I know I would like for getting a utility function out of me. If I or some simulated copy of me remained to actually be deciding things, I think I could get things I would not only like, but like and like liking. Especially if I can change myself from an insane ape who wishes it was a rationalist, to an actual rationalist through explicitly specified modifications guided by wished-for knowledge.

The best way I can think of to ensure that the extrapolated utility function is something like whatever is used in making my decisions, is to just use the brain circuits I already have that do that the way I like.

I also think a good idea might be to have a crowd of backup copies of me. One of us would try making some self-modifications in a sandboxed universe where their wishes could not get outside, and then the others would vote on whether to keep them.

Comment author: 28 September 2012 03:16:01AM 0 points [-]

Well, you don't prefer a world "where everyone else (excluding me and a few people I really like) is a second-class citizen to hot babes horrifies me." to the current world. If you can express such a judgement preferring one universe over another, and those judgements are transitive, you have a utility function.

And if you want to like liking them, that is also part of your utility function.

One confounding factor that you do bring up- the domain of one's utility function really doesn't need to include things outside the realm of possibility.

Comment author: 28 September 2012 06:18:48AM 0 points [-]

Well if deep down I wish that hot babes were given higher status in society than ugly females and males I didn't know (I'll roll with this example even though with omnipotence I could probably do things that would make this a nonissue) and I wish that I didn't wish it, and the genie maximizes my inferred utility function once (not maximizing it once and then maximizing the new one that doing so replaces it with), we would end up with a world where hot babes had higher status and I wished they didn't. The only thing I can see saving us from this would be if I also wanted my wants to be fulfilled, whatever they were. But then we would end up with me assigning Flimple utility to 2 + 2 being equal to 4.

I understand the argument "If you preferred the world ruled by hot babes it wouldn't horrify you," but what if that's just because when I imagine things at a global scale they turn featureless, and appearances don't matter to me anymore, but when I see the people in person they still do? What if, being able to see and socialize with every hot babe in the world changed my mind about whether I would want them to have higher social status, even if I was also able to see and socialize with everyone else?

What if the appearance of transitivity is only from drawing analogies from the last scenario I thought about (A) to the next one (B), but if I started a chain of analogies from somewhere else (C) my view on B would be completely different, such that you could take exchange rates of certain things I considered good in each scenario and construct a cycle of preference?

Comment author: 28 September 2012 07:03:31AM 0 points [-]

There's a difference between finding a global maximum of a function and finding the local derivative and moving infinity towards a local maximum.

Even if your actual utility function were for one group to have higher status, that does not imply that the greatest utility comes at the highers imbalance.

And if your preferences aren't transitive, you don't have a valid utility function. If you find yourself in a cycle of preference, you have probably failed to accurately judge two or more things which are hard to compare.

Comment author: 06 September 2012 02:25:04AM 1 point [-]

A much more real concern is that the genie is going to need to create and destroy at least BB(G) copies of you in order to produce the data you seek. Which is not good.