= 27d567bc2d48d9f40e9ccc20ea370b15)
CG_Morton comments on Rationality Quotes September 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1088)
Wish 1: "I wish for a paper containing the exact wording of a wish that, when spoken to you, would meet all my expectations for a wish granting X." For any value of X.
Wish 2: Profit.
Three wishes is overkill.
The scroll modifies your expectations. The genie twist-interprets X, and then assesses your expectations of the result of the genie's interpretation of X. ("Why, that's just what you'd expect destroying the world to do! What are you complaining about?") The complete list of expectations regarding X is at least slightly self-contradictory, so of course the genie has no option except to modify your expectations directly...
OOoh, is this now the "eliezer points out how your wish would go wrong" thread! I wanna play to! :p
"I wish for that which I'd wish for if I had an uninterrupted year of thinking about it and freely talking to a dedicated copy of Eliezer Yudovsky"
No sleep, or anything that would interrupt thinking about it, for a year, might lead to an interesting wish.
Well, it's obvious what happens then: the genie lets a dedicated copy of Eliezer out of a box.
Uh oh...
Eliezer Yudkowsky:
Eliezer Yudovsky:
The genie is, after all, all-powerful, so there are any number of subtle changes it could make that you didn't specify against that would immediately make you, or someone else, wish for the world to be destroyed. If that's the genie's goal, you have no chance. Heck, if it can choose it's form it could probably appear as some psycho-linguistic anomaly that hits your retina just right to make you into a person who would wish to end the world.
Really I'm just giving the genie a chance to show me that it's a nice guy. If it's super evil I'm doomed regardless, but this wish test (hopefully) distinguishes between a benevolent genie and one that's going to just be a dick.
If you consider three class of genies :
(A) a genie that's going to be "just be a dick" but is not skilled at it ;
(B) a genie that is benevolent ;
(C) a genie that's going to be "just be a dick" but is very skilled at it.
Your test will (may at least) tell apart A from (B or C). It won't tell apart B from C.
The "there is no safe wish" rule applies to C. Sure, if your genie is not skilled a being "evil" (having an utility function very different from yours), you can craft a wish that is beyond the genie's ability to twist it. But if the genie is skilled, much more intelligent than you are, with like the ability to spend the equivalent of one million of years of thinking how to twist the wish in one second, he'll find a flaw and use it.
"I wish for a paper containing the exact wording of a wish that, when spoken to you, would meet all my expectations as of September 3, 2012, for a wish granting X."
(Then, if my expectations yesterday did contain self-contradictions, the genie will do... whatever it did if I wished that 2 + 2 = 5.)
Genie provides a 3,000 foot long scroll, which if spoken perfectly will certainly do as you ask, but if spoken imperfectly in any of a million likely ways affords the genie room to screw you over.
Or the scroll is written in Martian.
I just take this as evidence that I -can't- beat the genie, and don't attempt any more wishes.
Whereas, if it's something simple then I have pretty strong evidence that the genie is -trying- to meet my wishes, that it's a benevolent genie.
"I wish for everything written on this scroll." Or some variation thereof that more exactly expresses that general idea.
All of the nouns named on the scroll appear. Some of them are things that the wording of the scroll expressly insists that the wish must avoid, due to their being lethal or otherwise undesirable.
"I wish for everything that would happen if I read this scroll perfectly."
Among other things, you would suffocate due to that four-minute no-breathing-allowed Martian word in paragraph nine.
4 minutes is survivable if trained.
Fine, thirty.
Not while speaking.
Ooooh. Well played.
In the likely event that it's impossible for you to read the scroll perfectly, it's true for all X that "X would happen if you read this scroll perfectly". Which means that anything the genie feels like doing satisfies that wish. Or possibly the genie has to make everything happen that could possibly happen. Neither of those seems like a good outcome.
Hm... how about "I wish to have all the skills and abilities required to formulate an unambiguous wish in standard English that would allow me to fulfill any of my non-contradictory desires that I choose, and to be able to choose which of any desires that are contradictory said wish would fulfill, and to be able to express that unambiguous wish in an unambiguous way in less than thirty seconds and with no consequences to incorrectly expressing that wish apart from the necessity of trying again to express it."
Wish 2: I wish for a text-to-speech device capable of reading from this scroll with perfect accuracy.
Wish 3: delegated to the device from #2.
Too bad Martian words sound exactly like lethal sonic weapons and your original X that your wish is about doesn't, strictly speaking, require resurrecting you to enjoy it.
Or, the genie doesn't have to respond to wishes that don't come out of Master's mouth.
Are we going to keep patching up every hole she points out? Or admit that a UFAI genie can be smarter than any human (even if that human is our esteemed Alicorn, or (gasp!) Eliezer)?
Text-to-speech device provided. It reads from the scroll with perfect accuracy and low speed. It will take a few hundred years to complete this task.
You will need to change the batteries once an hour; it you forget, it starts reading from the start of the scroll again. (And where do you get a large supply of size Q batteries, in any case?)
I know some electrical engineers. It's not all that hard to rig up an uniniterruptible power supply that runs off line voltage. The delay is inconvenient, but for the right wish it's acceptable.
I'm pretty sure your belief network is not coherent enough so that it is possible to "meet all your expectations", there must be somewhere two expectations which you hold but which aren't, in fact, compatible. So the wish will fizzle ;)
A wish is a pretty constrained thing, for some wishes.
If I wish for a pile of gold, my expectations probably constrain lots of externalities like 'Nobody is hurt acquiring the gold, it isn't taken from somewhere else, it is simply generated and deposited at my feet, but not, like, crushing me, or using the molecules of my body as raw material, or really anything that kills me for that matter'. Mostly my expectations are about things that won't happen, not things that will happen that might conflict (that consists only of: the gold will appear before me and will be real and persistent).
If you try this with a wish for world peace, you're likely to get screwed. But I think that's a given no matter your strategy. Don't wish for goals, wish for the tools to achieve your goals - you'll probably be more satisfied with the result to boot.
You're already lowering your claim, it's not longer "for any value of X".
But even so...
"Nobody is hurt acquiring the gold" does that include people hurt because your sudden new gold decrease the market value of gold, so people owning stocks of gold or speculating on an increase of the gold price are hurt ? Sure, you can say "it's insignificant", but how will a genie tell that apart ? Your expectation of what having a sudden supply of gold on the market would do and the reality of how it'll unfold probably don't match. So the genie will have to do corrections for that... which will themselves have side-effects...
Also, you'll probably realize once you've some gold that gold doesn't bring you as much as you thought it would bring you (at least, it happens to most lottery winner), so even if you genuinely get the gold, it'll fail to "meet all your expectations" of having gold. Unless the genie also fixes you so you get as much utility/happiness/... from the gold as you expected to get from it. And as soon as the genie has to start fixing you... game over.
I simplify here because a lot of people think I will have contradictory expectations for a more complex event.
But I think you're being even more picky here. Do I -expect- that increasing the amount of gold in the world will slightly affect the market value? Yes. But I haven't wished anything related to that, my wish is -only- about some gold appearing in front of me.
Having the genie magically change how much utility I get from the gold is an even more ridiculous extension. If I wish for gold, why the heck would the genie feel it was his job to change my mental state to make me like gold more?
Possibly we just think very differently, and your 'expectation' of what would happen when gold appears also includes every thing you would do with that gold later, despite, among many, many things, not even knowing -when- you would speak the wish to get the gold, or what form it would appear in. And you even have in mind some specific level of happiness that you 'expect' to get from it. If so, you're right, this trick will not work for you.
If you wish for gold, it's because you have expectation on what you'll do with that gold. Maybe fuzzy ones, but if you didn't have any, you wouldn't wish for gold. So you can't dissociate the gold and the use when what you're speaking about is "expectations".
Or else, solutions like "the world is changed so that the precious metal is lead, and gold has low value, but all the rest is the same" would work. And that wouldn't meet your "expectations" about the wish at all.
An altogether fairly harmless outcome.
Couldn't you just wish that all your expectations for a wish granting X were granted, and take out the second step?
This presumes, of course, that my expectations for a wish granting X, for some value of X, is such that having a wish granted that meets them is profitable.