Vivid comments on Rationality Quotes September 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1088)
Wish 1: "I wish for a paper containing the exact wording of a wish that, when spoken to you, would meet all my expectations for a wish granting X." For any value of X.
Wish 2: Profit.
Three wishes is overkill.
I'm pretty sure your belief network is not coherent enough so that it is possible to "meet all your expectations", there must be somewhere two expectations which you hold but which aren't, in fact, compatible. So the wish will fizzle ;)
A wish is a pretty constrained thing, for some wishes.
If I wish for a pile of gold, my expectations probably constrain lots of externalities like 'Nobody is hurt acquiring the gold, it isn't taken from somewhere else, it is simply generated and deposited at my feet, but not, like, crushing me, or using the molecules of my body as raw material, or really anything that kills me for that matter'. Mostly my expectations are about things that won't happen, not things that will happen that might conflict (that consists only of: the gold will appear before me and will be real and persistent).
If you try this with a wish for world peace, you're likely to get screwed. But I think that's a given no matter your strategy. Don't wish for goals, wish for the tools to achieve your goals - you'll probably be more satisfied with the result to boot.
You're already lowering your claim, it's not longer "for any value of X".
But even so...
"Nobody is hurt acquiring the gold" does that include people hurt because your sudden new gold decrease the market value of gold, so people owning stocks of gold or speculating on an increase of the gold price are hurt ? Sure, you can say "it's insignificant", but how will a genie tell that apart ? Your expectation of what having a sudden supply of gold on the market would do and the reality of how it'll unfold probably don't match. So the genie will have to do corrections for that... which will themselves have side-effects...
Also, you'll probably realize once you've some gold that gold doesn't bring you as much as you thought it would bring you (at least, it happens to most lottery winner), so even if you genuinely get the gold, it'll fail to "meet all your expectations" of having gold. Unless the genie also fixes you so you get as much utility/happiness/... from the gold as you expected to get from it. And as soon as the genie has to start fixing you... game over.
I simplify here because a lot of people think I will have contradictory expectations for a more complex event.
But I think you're being even more picky here. Do I -expect- that increasing the amount of gold in the world will slightly affect the market value? Yes. But I haven't wished anything related to that, my wish is -only- about some gold appearing in front of me.
Having the genie magically change how much utility I get from the gold is an even more ridiculous extension. If I wish for gold, why the heck would the genie feel it was his job to change my mental state to make me like gold more?
Possibly we just think very differently, and your 'expectation' of what would happen when gold appears also includes every thing you would do with that gold later, despite, among many, many things, not even knowing -when- you would speak the wish to get the gold, or what form it would appear in. And you even have in mind some specific level of happiness that you 'expect' to get from it. If so, you're right, this trick will not work for you.
If you wish for gold, it's because you have expectation on what you'll do with that gold. Maybe fuzzy ones, but if you didn't have any, you wouldn't wish for gold. So you can't dissociate the gold and the use when what you're speaking about is "expectations".
Or else, solutions like "the world is changed so that the precious metal is lead, and gold has low value, but all the rest is the same" would work. And that wouldn't meet your "expectations" about the wish at all.
An altogether fairly harmless outcome.
Genie provides a 3,000 foot long scroll, which if spoken perfectly will certainly do as you ask, but if spoken imperfectly in any of a million likely ways affords the genie room to screw you over.
Or the scroll is written in Martian.
Wish 2: I wish for a text-to-speech device capable of reading from this scroll with perfect accuracy.
Wish 3: delegated to the device from #2.
Are we going to keep patching up every hole she points out? Or admit that a UFAI genie can be smarter than any human (even if that human is our esteemed Alicorn, or (gasp!) Eliezer)?
Text-to-speech device provided. It reads from the scroll with perfect accuracy and low speed. It will take a few hundred years to complete this task.
You will need to change the batteries once an hour; it you forget, it starts reading from the start of the scroll again. (And where do you get a large supply of size Q batteries, in any case?)
I know some electrical engineers. It's not all that hard to rig up an uniniterruptible power supply that runs off line voltage. The delay is inconvenient, but for the right wish it's acceptable.
Too bad Martian words sound exactly like lethal sonic weapons and your original X that your wish is about doesn't, strictly speaking, require resurrecting you to enjoy it.
Or, the genie doesn't have to respond to wishes that don't come out of Master's mouth.
"I wish for everything written on this scroll." Or some variation thereof that more exactly expresses that general idea.
All of the nouns named on the scroll appear. Some of them are things that the wording of the scroll expressly insists that the wish must avoid, due to their being lethal or otherwise undesirable.
"I wish for everything that would happen if I read this scroll perfectly."
Among other things, you would suffocate due to that four-minute no-breathing-allowed Martian word in paragraph nine.
4 minutes is survivable if trained.
Fine, thirty.
Not while speaking.
Ooooh. Well played.
In the likely event that it's impossible for you to read the scroll perfectly, it's true for all X that "X would happen if you read this scroll perfectly". Which means that anything the genie feels like doing satisfies that wish. Or possibly the genie has to make everything happen that could possibly happen. Neither of those seems like a good outcome.
Hm... how about "I wish to have all the skills and abilities required to formulate an unambiguous wish in standard English that would allow me to fulfill any of my non-contradictory desires that I choose, and to be able to choose which of any desires that are contradictory said wish would fulfill, and to be able to express that unambiguous wish in an unambiguous way in less than thirty seconds and with no consequences to incorrectly expressing that wish apart from the necessity of trying again to express it."
I just take this as evidence that I -can't- beat the genie, and don't attempt any more wishes.
Whereas, if it's something simple then I have pretty strong evidence that the genie is -trying- to meet my wishes, that it's a benevolent genie.
Couldn't you just wish that all your expectations for a wish granting X were granted, and take out the second step?
The scroll modifies your expectations. The genie twist-interprets X, and then assesses your expectations of the result of the genie's interpretation of X. ("Why, that's just what you'd expect destroying the world to do! What are you complaining about?") The complete list of expectations regarding X is at least slightly self-contradictory, so of course the genie has no option except to modify your expectations directly...
OOoh, is this now the "eliezer points out how your wish would go wrong" thread! I wanna play to! :p
"I wish for that which I'd wish for if I had an uninterrupted year of thinking about it and freely talking to a dedicated copy of Eliezer Yudovsky"
Uh oh...
Eliezer Yudkowsky:
Eliezer Yudovsky:
No sleep, or anything that would interrupt thinking about it, for a year, might lead to an interesting wish.
Well, it's obvious what happens then: the genie lets a dedicated copy of Eliezer out of a box.
The genie is, after all, all-powerful, so there are any number of subtle changes it could make that you didn't specify against that would immediately make you, or someone else, wish for the world to be destroyed. If that's the genie's goal, you have no chance. Heck, if it can choose it's form it could probably appear as some psycho-linguistic anomaly that hits your retina just right to make you into a person who would wish to end the world.
Really I'm just giving the genie a chance to show me that it's a nice guy. If it's super evil I'm doomed regardless, but this wish test (hopefully) distinguishes between a benevolent genie and one that's going to just be a dick.
If you consider three class of genies :
(A) a genie that's going to be "just be a dick" but is not skilled at it ;
(B) a genie that is benevolent ;
(C) a genie that's going to be "just be a dick" but is very skilled at it.
Your test will (may at least) tell apart A from (B or C). It won't tell apart B from C.
The "there is no safe wish" rule applies to C. Sure, if your genie is not skilled a being "evil" (having an utility function very different from yours), you can craft a wish that is beyond the genie's ability to twist it. But if the genie is skilled, much more intelligent than you are, with like the ability to spend the equivalent of one million of years of thinking how to twist the wish in one second, he'll find a flaw and use it.
"I wish for a paper containing the exact wording of a wish that, when spoken to you, would meet all my expectations as of September 3, 2012, for a wish granting X."
(Then, if my expectations yesterday did contain self-contradictions, the genie will do... whatever it did if I wished that 2 + 2 = 5.)
This presumes, of course, that my expectations for a wish granting X, for some value of X, is such that having a wish granted that meets them is profitable.
Er... actually the genie is offering at most two rounds of feedback.
Sorry about the pedantry, it's just that as a professional specialist in genies I have a tendency to notice that sort of thing.
Rather than a technical correction you seem just to be substituting a different meaning of 'feedback'. The author would certainly not agree that "You get 0 feedback from 1 wish".
Mind you I am wary of the the fundamental message of the quote. Feedback? One of the most obviously important purposes of getting feedback is to avoid catastrophic failure. Yet catastrophic failures are exactly the kind of thing that will prevent you from using the next wish. So this is "Just Feedback" that can Kill You Off For Real despite the miraculous intervention you have access to.
I'd say "What the genie is really offering is a wish and two chances to change your mind---assuming you happen to be still alive and capable of constructing corrective wishes".
0 feedback is exactly what you get from 1 wish. "Feedback" isn't just information, it's something that can control a system's future behavior - so unless you expect to find another genie bottle later, "Finding out how your wish worked" isn't the same as feedback at all.
I think it was clear that I inferred this as the new definition you were trying to substitute. I was very nearly as impressed as if you 'corrected' him by telling him that it isn't "feedback" if nobody is around to hear it, or perhaps told him that oxygen is a metal.
...or unless genies granting wishes is actually part of the same system as the larger world, such that what I learn from the results of a wish can be applied (by me or some other observer) to better calibrate expectations from other actions in that system besides wishing-from-genies.
One well-known folk tale is based on precisely this interpretation. Probably more than one.
Why only 2 rounds of feedback if you have 3 wishes?
The third one's for keeps: you can't wish the consequences away.
Right, but the consequences still qualify as feedback, no?
I always imagine the genie just goes back into his lamp to sleep or whatever, so in the hypothetical as it exists in my head, no. But I guess there could be a highly ambitious Genie looking for feedback after your last wish, so maybe.
I think in this case, Eliezer in talking about a genie like in Failed Utopia 4-2 who grants his wish, and then keeps working, ignoring feedback, because he just doesn't care, because caring isn't part of the wish.
The genie doesn't care about consequences, he just cares about the wishes. The second wish and third wish are the feedback.
The feedback is for you, not what you happen to say to the genie.
I should like to point out that anyone in this situation who wishes what would've been their first wish if they had three wishes is a bloody idiot.
"anyone in this situation" who believes that an elderly woman before him can grant arbitrary wishes is a bloody idiot to begin with, so the bar is set low.
That's not what's going on though. The traveller is assuming, reasonably, that his third wish is reversing the amnesiac effects of his second. He's not just starting fr om scratch.
I don't think this follows from the text. The hag tells him "but second wish was for me to return everything to the way it was before you had made your first wish. That's why you remember nothing; because everything is the way it was before you made any wishes".
So she told him that he had been an amnesiac before any wishes were granted. Therefore he should have already guessed that his first wish was to know who he was -- and that this proved a bad idea, since his second wish was to reverse the first.
So: A genie pops up and says, "You have one wish left."
What do you wish for? Because presumably the giftwrapped FAI didn't work so great.
I bet he'd wish "to erase all uFAI from existence before they're even born. Every uFAI in every universe, from the past and the future, with my own hands."
Nobody believes in the future.
Nobody accepts the future.
Then -
Perhaps I'm simply being an idiot, but ... huh?
"I wish to know what went wrong with my first wish."
This way, I at least end up with improved knowledge of what to avoid in the future.
Alternatively, "I wish for a magical map, which shows me, in real time, the location of every trapped genie and other potential source of wishes in the world." Depending on how many there are, I can potentially get a lot more feedback that way.
"I wish for this wish to have no further effect beyond this utterance."
Overwhelmingly probable dire consequence: You and everyone you love dies (over a period of 70 years) then, eventually, your entire species goes extinct. But hey, at least it's not "your fault".
You can't use that tool to solve that problem.
Meanwhile, you have <= 70 years to solve it another way.
But, alas, it's the wish that maximizes my expected utility -- for the malicious genie, anyway.
"Destroy yourself as near to immediately as possible, given that your method of self destruction causes no avoidable harm to anything larger than an ant."
They shrink the planet down to below our Schwarzschild radius, holding spacetime in place for just long enough to explain what you just did.
Alternately, they declare your wish is logically contradictory - genies are larger than ants.
At the start of the scenario, you are already dead with probability approaching 1. Trying to knock the gun away can't hurt.
It should be noted that night hags are sufficiently smart, powerful, and evil that your best case scenario upon meeting one is a quick and painful death.
But not everything is the way it was. Before he made any wishes, he had three.
She missed the chance to trap him in an infinite loop.
But then the Hag would be trapped too.
She gets delight from tormenting mortals, but tormenting the same one, in the same way eternally, would probably be too close to wireheading for her.
Well, if she got bored, she could experiment with different ways to present his wishes to him at the "beginning" and see if she can get him to wish for something to else, or word it a bit differently. Since she seems to retain memories of the whole thing. (Which is again, things not being how they were, but.)
The psuedo-meta-textual answer is that Morte is lying to Yves while the main character overhears. Morte's making up the story just to mess around with him.
Background information is that gur znva punenpgre znqr n qrny jvgu n Unt (rivy cneg-gvzr travr), tnvavat vzzbegnyvgl naq nzarfvn. At the start of the story, the main character somehow broke out of an infinite loop of torture; he's stopped having Anterograde Amnesia, but still cannot remember much from before the cycle broke, and is on a quest to remember who he is. Morte is trying to dissuade the main character from finding out who he is, showing that things can be terrible even without an infinite loop.
Now that would be evil.
If his first wished disappeared him forever, how did he ever get a second wish?
Apparently I suck at reading.
... which one wish, carefully phrased, could also provide.
you can't wish for more wishes
but does "I wish my wishes were divided up into 100 rounds of feedback each, each roughly equivalent to 1/100 of a wish" fall under that?
My impression would be that you could use one wish to divide a second wish up in such a manner. Using it to divide itself up would 'fall under'. (I'm not sure how the genie would resolve the wish into a practical outcome.)
Yea, what it does is give you 200 subwishes, not 300.
"I wish for the result of the hypothetical nth wish I would make if I was allowed to make n wishes in the limit as n went to infinity each time believing that the next wish would be my only one and all previous wishes would be reversed, or if that limit does not exist, pick n = busy beaver function of Graham's number."
I can see a genie taking a shortcut here.
— Granny Weatherwax, A Hat Full of Sky.
In short, the genie may well conclude that every m'th wish, for some m (Granny Weatherwax suggests here that 'm' is three) your wish would be to have never met the genie in the first place. At this point, if you're lucky, the genie will use a value of n that's a multiple of m. If you're unlucky, the genie will use a value of n that's km-1 for some integer k...
Alternatively, you'll end up with a genie who can't handle the math and does not understand what you're asking for.
I'm pretty sure this would result in the genie killing you.
A much more real concern is that the genie is going to need to create and destroy at least BB(G) copies of you in order to produce the data you seek. Which is not good.
Because I would wish to kill myself eventually? It's hard to imagine that I would do that, faced with unlimited wishes. If I got bored, I could just wish the boredom away.
Though on reflection this wish needs a safeguard against infinite recursion, and a bliss-universe for any simulated copies of me the genie creates to determine what my wishes would be.
In a sufficiently bad situation, you may wish for the genie to kill you because you think that's your only wish. It's not likely for any given wish, but would happen eventually (and ends the recursion, so that's one of the few stable wishes).
If I kill myself, there is no nth wish as n -> infinity, or a busy beaver function of Graham's numberth wish, so the first wish is wishing for something undefined.
Also, the probability of any of the individually improbable events where I kill myself happening is bounded above by the some of their probabilities, and they could be a convergent infinite series, if the probability of wanting to kill myself went down each time. Even though I stipulated that it's if I believed each wish was the last, I might do something like "except don't grant this wish if it would result in me wanting to kill myself or dying before I could consider the question" in each hypothetical wish. Or grant myself superintelligence as part of one of the hypothetical wishes, and come up with an even better safeguard when I found myself (to my great irrational surprise) getting another wish.
There is not even necessarily a tiny chance of wanting to kill myself. Good epistemology says to think there is, just in case, but some things are actually impossible. Using wishes to make it impossible for me to want to kill myself might come faster than killing myself.
I think you're right, though I'm not sure that's exactly a good thing.
I see no particular reason to expect that to be the case.
Excellent point. That might just work (though I'm sure there are still a thousand ways it could go mind-bogglingly wrong).
If you did eventually wish for death, then you would have judged that death is the best thing you can wish for, after having tried as many alternatives as possible.
Are you going to try to use your prior (I don't want to die) to argue with your future self who has experienced the results (and determines that you would be happier dying right now than getting any wish)?
I would not want to kill myself if my distant-future self wanted to die or wanted me to die immediately. I think it is much more likely that I would accidentally self-modify in a manner I wouldn't like if I reflected on it now and that would lead to wishing for death than that my current self with additional knowledge would chose death over omnipotence.
I don't think the factual question "would I be happier dying right now" would necessarily be the one that decided the policy question of "will I chose do die" for both me and my future self, because we could each care about different things.
And with a warning "The way things are going, you'll end up wanting to die." I could change my wishes and maybe get something better.
"Does my current utility function contain a global maximum at the case where I wish for and receive death right now?" is a really scary question to ask a genie.
I would prefer "I wish for the world to changed in such a manner as my present actual utility function, (explicitly distinct from my perception of my utility function), is at the global maximum possible without altering my present actual utility function."
Or in colloquial language "Give me what I want, not what I think I want."
That sounds pretty scary too. I don't think I am close enough to being an agent to have a well-defined utility function. If I do (paradoxical as it sounds), it's probably not something I would reflectively like. For example, I think I have more empathy for things I am sexually attracted to. But the idea of a world where everyone else (excluding me and a few people I really like) is a second-class citizen to hot babes horrifies me. But with the wrong kind of extrapolation, I bet that could be said to be what I want.
I can't easily describe any procedure I know I would like for getting a utility function out of me. If I or some simulated copy of me remained to actually be deciding things, I think I could get things I would not only like, but like and like liking. Especially if I can change myself from an insane ape who wishes it was a rationalist, to an actual rationalist through explicitly specified modifications guided by wished-for knowledge.
The best way I can think of to ensure that the extrapolated utility function is something like whatever is used in making my decisions, is to just use the brain circuits I already have that do that the way I like.
I also think a good idea might be to have a crowd of backup copies of me. One of us would try making some self-modifications in a sandboxed universe where their wishes could not get outside, and then the others would vote on whether to keep them.
Kate Evans https://twitter.com/aristosophy/status/240589485098795008
How about wishing for enough judgment to make wishes that you won't regret? Additional clause: the genie isn't allowed to deteriorate your mental capacities below their current level.