= 27d567bc2d48d9f40e9ccc20ea370b15)
Eliezer_Yudkowsky comments on Rationality Quotes September 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1088)
Er... actually the genie is offering at most two rounds of feedback.
Sorry about the pedantry, it's just that as a professional specialist in genies I have a tendency to notice that sort of thing.
Rather than a technical correction you seem just to be substituting a different meaning of 'feedback'. The author would certainly not agree that "You get 0 feedback from 1 wish".
Mind you I am wary of the the fundamental message of the quote. Feedback? One of the most obviously important purposes of getting feedback is to avoid catastrophic failure. Yet catastrophic failures are exactly the kind of thing that will prevent you from using the next wish. So this is "Just Feedback" that can Kill You Off For Real despite the miraculous intervention you have access to.
I'd say "What the genie is really offering is a wish and two chances to change your mind---assuming you happen to be still alive and capable of constructing corrective wishes".
One well-known folk tale is based on precisely this interpretation. Probably more than one.
0 feedback is exactly what you get from 1 wish. "Feedback" isn't just information, it's something that can control a system's future behavior - so unless you expect to find another genie bottle later, "Finding out how your wish worked" isn't the same as feedback at all.
...or unless genies granting wishes is actually part of the same system as the larger world, such that what I learn from the results of a wish can be applied (by me or some other observer) to better calibrate expectations from other actions in that system besides wishing-from-genies.
I think it was clear that I inferred this as the new definition you were trying to substitute. I was very nearly as impressed as if you 'corrected' him by telling him that it isn't "feedback" if nobody is around to hear it, or perhaps told him that oxygen is a metal.
Why only 2 rounds of feedback if you have 3 wishes?
The third one's for keeps: you can't wish the consequences away.
I should like to point out that anyone in this situation who wishes what would've been their first wish if they had three wishes is a bloody idiot.
So: A genie pops up and says, "You have one wish left."
What do you wish for? Because presumably the giftwrapped FAI didn't work so great.
"I wish to know what went wrong with my first wish."
This way, I at least end up with improved knowledge of what to avoid in the future.
Alternatively, "I wish for a magical map, which shows me, in real time, the location of every trapped genie and other potential source of wishes in the world." Depending on how many there are, I can potentially get a lot more feedback that way.
I bet he'd wish "to erase all uFAI from existence before they're even born. Every uFAI in every universe, from the past and the future, with my own hands."
Nobody believes in the future.
Nobody accepts the future.
Then -
Perhaps I'm simply being an idiot, but ... huh?
"I wish for this wish to have no further effect beyond this utterance."
Overwhelmingly probable dire consequence: You and everyone you love dies (over a period of 70 years) then, eventually, your entire species goes extinct. But hey, at least it's not "your fault".
But, alas, it's the wish that maximizes my expected utility -- for the malicious genie, anyway.
You can't use that tool to solve that problem.
Meanwhile, you have <= 70 years to solve it another way.
"Destroy yourself as near to immediately as possible, given that your method of self destruction causes no avoidable harm to anything larger than an ant."
They shrink the planet down to below our Schwarzschild radius, holding spacetime in place for just long enough to explain what you just did.
Alternately, they declare your wish is logically contradictory - genies are larger than ants.
At the start of the scenario, you are already dead with probability approaching 1. Trying to knock the gun away can't hurt.
That's not what's going on though. The traveller is assuming, reasonably, that his third wish is reversing the amnesiac effects of his second. He's not just starting fr om scratch.
I don't think this follows from the text. The hag tells him "but second wish was for me to return everything to the way it was before you had made your first wish. That's why you remember nothing; because everything is the way it was before you made any wishes".
So she told him that he had been an amnesiac before any wishes were granted. Therefore he should have already guessed that his first wish was to know who he was -- and that this proved a bad idea, since his second wish was to reverse the first.
"anyone in this situation" who believes that an elderly woman before him can grant arbitrary wishes is a bloody idiot to begin with, so the bar is set low.
It should be noted that night hags are sufficiently smart, powerful, and evil that your best case scenario upon meeting one is a quick and painful death.
But not everything is the way it was. Before he made any wishes, he had three.
She missed the chance to trap him in an infinite loop.
But then the Hag would be trapped too.
She gets delight from tormenting mortals, but tormenting the same one, in the same way eternally, would probably be too close to wireheading for her.
Well, if she got bored, she could experiment with different ways to present his wishes to him at the "beginning" and see if she can get him to wish for something to else, or word it a bit differently. Since she seems to retain memories of the whole thing. (Which is again, things not being how they were, but.)
The psuedo-meta-textual answer is that Morte is lying to Yves while the main character overhears. Morte's making up the story just to mess around with him.
Background information is that gur znva punenpgre znqr n qrny jvgu n Unt (rivy cneg-gvzr travr), tnvavat vzzbegnyvgl naq nzarfvn. At the start of the story, the main character somehow broke out of an infinite loop of torture; he's stopped having Anterograde Amnesia, but still cannot remember much from before the cycle broke, and is on a quest to remember who he is. Morte is trying to dissuade the main character from finding out who he is, showing that things can be terrible even without an infinite loop.
Now that would be evil.
If his first wished disappeared him forever, how did he ever get a second wish?
Apparently I suck at reading.
Right, but the consequences still qualify as feedback, no?
I always imagine the genie just goes back into his lamp to sleep or whatever, so in the hypothetical as it exists in my head, no. But I guess there could be a highly ambitious Genie looking for feedback after your last wish, so maybe.
I think in this case, Eliezer in talking about a genie like in Failed Utopia 4-2 who grants his wish, and then keeps working, ignoring feedback, because he just doesn't care, because caring isn't part of the wish.
The genie doesn't care about consequences, he just cares about the wishes. The second wish and third wish are the feedback.
The feedback is for you, not what you happen to say to the genie.