= 27d567bc2d48d9f40e9ccc20ea370b15)
Ezekiel comments on Rationality Quotes September 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1088)
... which one wish, carefully phrased, could also provide.
you can't wish for more wishes
"I wish for the result of the hypothetical nth wish I would make if I was allowed to make n wishes in the limit as n went to infinity each time believing that the next wish would be my only one and all previous wishes would be reversed, or if that limit does not exist, pick n = busy beaver function of Graham's number."
I can see a genie taking a shortcut here.
— Granny Weatherwax, A Hat Full of Sky.
In short, the genie may well conclude that every m'th wish, for some m (Granny Weatherwax suggests here that 'm' is three) your wish would be to have never met the genie in the first place. At this point, if you're lucky, the genie will use a value of n that's a multiple of m. If you're unlucky, the genie will use a value of n that's km-1 for some integer k...
Alternatively, you'll end up with a genie who can't handle the math and does not understand what you're asking for.
I'm pretty sure this would result in the genie killing you.
Because I would wish to kill myself eventually? It's hard to imagine that I would do that, faced with unlimited wishes. If I got bored, I could just wish the boredom away.
Though on reflection this wish needs a safeguard against infinite recursion, and a bliss-universe for any simulated copies of me the genie creates to determine what my wishes would be.
In a sufficiently bad situation, you may wish for the genie to kill you because you think that's your only wish. It's not likely for any given wish, but would happen eventually (and ends the recursion, so that's one of the few stable wishes).
If I kill myself, there is no nth wish as n -> infinity, or a busy beaver function of Graham's numberth wish, so the first wish is wishing for something undefined.
Also, the probability of any of the individually improbable events where I kill myself happening is bounded above by the some of their probabilities, and they could be a convergent infinite series, if the probability of wanting to kill myself went down each time. Even though I stipulated that it's if I believed each wish was the last, I might do something like "except don't grant this wish if it would result in me wanting to kill myself or dying before I could consider the question" in each hypothetical wish. Or grant myself superintelligence as part of one of the hypothetical wishes, and come up with an even better safeguard when I found myself (to my great irrational surprise) getting another wish.
There is not even necessarily a tiny chance of wanting to kill myself. Good epistemology says to think there is, just in case, but some things are actually impossible. Using wishes to make it impossible for me to want to kill myself might come faster than killing myself.
I think you're right, though I'm not sure that's exactly a good thing.
I see no particular reason to expect that to be the case.
Excellent point. That might just work (though I'm sure there are still a thousand ways it could go mind-bogglingly wrong).
If you did eventually wish for death, then you would have judged that death is the best thing you can wish for, after having tried as many alternatives as possible.
Are you going to try to use your prior (I don't want to die) to argue with your future self who has experienced the results (and determines that you would be happier dying right now than getting any wish)?
I would not want to kill myself if my distant-future self wanted to die or wanted me to die immediately. I think it is much more likely that I would accidentally self-modify in a manner I wouldn't like if I reflected on it now and that would lead to wishing for death than that my current self with additional knowledge would chose death over omnipotence.
I don't think the factual question "would I be happier dying right now" would necessarily be the one that decided the policy question of "will I chose do die" for both me and my future self, because we could each care about different things.
And with a warning "The way things are going, you'll end up wanting to die." I could change my wishes and maybe get something better.
"Does my current utility function contain a global maximum at the case where I wish for and receive death right now?" is a really scary question to ask a genie.
I would prefer "I wish for the world to changed in such a manner as my present actual utility function, (explicitly distinct from my perception of my utility function), is at the global maximum possible without altering my present actual utility function."
Or in colloquial language "Give me what I want, not what I think I want."
That sounds pretty scary too. I don't think I am close enough to being an agent to have a well-defined utility function. If I do (paradoxical as it sounds), it's probably not something I would reflectively like. For example, I think I have more empathy for things I am sexually attracted to. But the idea of a world where everyone else (excluding me and a few people I really like) is a second-class citizen to hot babes horrifies me. But with the wrong kind of extrapolation, I bet that could be said to be what I want.
I can't easily describe any procedure I know I would like for getting a utility function out of me. If I or some simulated copy of me remained to actually be deciding things, I think I could get things I would not only like, but like and like liking. Especially if I can change myself from an insane ape who wishes it was a rationalist, to an actual rationalist through explicitly specified modifications guided by wished-for knowledge.
The best way I can think of to ensure that the extrapolated utility function is something like whatever is used in making my decisions, is to just use the brain circuits I already have that do that the way I like.
I also think a good idea might be to have a crowd of backup copies of me. One of us would try making some self-modifications in a sandboxed universe where their wishes could not get outside, and then the others would vote on whether to keep them.
A much more real concern is that the genie is going to need to create and destroy at least BB(G) copies of you in order to produce the data you seek. Which is not good.
but does "I wish my wishes were divided up into 100 rounds of feedback each, each roughly equivalent to 1/100 of a wish" fall under that?
My impression would be that you could use one wish to divide a second wish up in such a manner. Using it to divide itself up would 'fall under'. (I'm not sure how the genie would resolve the wish into a practical outcome.)
Yea, what it does is give you 200 subwishes, not 300.