Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: kilobug 07 September 2012 02:29:16PM 1 point [-]

You're already lowering your claim, it's not longer "for any value of X".

But even so...

"Nobody is hurt acquiring the gold" does that include people hurt because your sudden new gold decrease the market value of gold, so people owning stocks of gold or speculating on an increase of the gold price are hurt ? Sure, you can say "it's insignificant", but how will a genie tell that apart ? Your expectation of what having a sudden supply of gold on the market would do and the reality of how it'll unfold probably don't match. So the genie will have to do corrections for that... which will themselves have side-effects...

Also, you'll probably realize once you've some gold that gold doesn't bring you as much as you thought it would bring you (at least, it happens to most lottery winner), so even if you genuinely get the gold, it'll fail to "meet all your expectations" of having gold. Unless the genie also fixes you so you get as much utility/happiness/... from the gold as you expected to get from it. And as soon as the genie has to start fixing you... game over.

Comment author: CG_Morton 07 September 2012 03:21:23PM 1 point [-]

I simplify here because a lot of people think I will have contradictory expectations for a more complex event.

But I think you're being even more picky here. Do I -expect- that increasing the amount of gold in the world will slightly affect the market value? Yes. But I haven't wished anything related to that, my wish is -only- about some gold appearing in front of me.

Having the genie magically change how much utility I get from the gold is an even more ridiculous extension. If I wish for gold, why the heck would the genie feel it was his job to change my mental state to make me like gold more?

Possibly we just think very differently, and your 'expectation' of what would happen when gold appears also includes every thing you would do with that gold later, despite, among many, many things, not even knowing -when- you would speak the wish to get the gold, or what form it would appear in. And you even have in mind some specific level of happiness that you 'expect' to get from it. If so, you're right, this trick will not work for you.

Comment author: Eliezer_Yudkowsky 04 September 2012 09:17:05PM 9 points [-]

The scroll modifies your expectations. The genie twist-interprets X, and then assesses your expectations of the result of the genie's interpretation of X. ("Why, that's just what you'd expect destroying the world to do! What are you complaining about?") The complete list of expectations regarding X is at least slightly self-contradictory, so of course the genie has no option except to modify your expectations directly...

Comment author: CG_Morton 07 September 2012 02:10:23PM 1 point [-]

The genie is, after all, all-powerful, so there are any number of subtle changes it could make that you didn't specify against that would immediately make you, or someone else, wish for the world to be destroyed. If that's the genie's goal, you have no chance. Heck, if it can choose it's form it could probably appear as some psycho-linguistic anomaly that hits your retina just right to make you into a person who would wish to end the world.

Really I'm just giving the genie a chance to show me that it's a nice guy. If it's super evil I'm doomed regardless, but this wish test (hopefully) distinguishes between a benevolent genie and one that's going to just be a dick.

Comment author: kilobug 04 September 2012 06:28:57PM 4 points [-]

I'm pretty sure your belief network is not coherent enough so that it is possible to "meet all your expectations", there must be somewhere two expectations which you hold but which aren't, in fact, compatible. So the wish will fizzle ;)

Comment author: CG_Morton 07 September 2012 02:01:34PM 1 point [-]

A wish is a pretty constrained thing, for some wishes.

If I wish for a pile of gold, my expectations probably constrain lots of externalities like 'Nobody is hurt acquiring the gold, it isn't taken from somewhere else, it is simply generated and deposited at my feet, but not, like, crushing me, or using the molecules of my body as raw material, or really anything that kills me for that matter'. Mostly my expectations are about things that won't happen, not things that will happen that might conflict (that consists only of: the gold will appear before me and will be real and persistent).

If you try this with a wish for world peace, you're likely to get screwed. But I think that's a given no matter your strategy. Don't wish for goals, wish for the tools to achieve your goals - you'll probably be more satisfied with the result to boot.

Comment author: Alicorn 04 September 2012 06:29:23PM 14 points [-]

Genie provides a 3,000 foot long scroll, which if spoken perfectly will certainly do as you ask, but if spoken imperfectly in any of a million likely ways affords the genie room to screw you over.

Or the scroll is written in Martian.

Comment author: CG_Morton 07 September 2012 01:53:35PM 4 points [-]

I just take this as evidence that I -can't- beat the genie, and don't attempt any more wishes.

Whereas, if it's something simple then I have pretty strong evidence that the genie is -trying- to meet my wishes, that it's a benevolent genie.

Comment author: Vivid 01 September 2012 09:33:16AM *  36 points [-]

One wish can achieve as much as you want. What the genie is really offering is three rounds of feedback.

Aristosophy

WIll Newsome is such a badass.

Plato

Comment author: CG_Morton 04 September 2012 05:27:55PM 3 points [-]

Wish 1: "I wish for a paper containing the exact wording of a wish that, when spoken to you, would meet all my expectations for a wish granting X." For any value of X.

Wish 2: Profit.

Three wishes is overkill.

Comment author: Dmytry 11 January 2012 09:46:16PM *  2 points [-]

Well, we don't blame special relativity for seeming to fail with all the thought experiments involving objects moving faster than speed of light. edit: that is to say, it is important that thought experiments remain within certain bounds. In the case of the trolley problems, the small difference between the assumptions and real world (neglecting the small false positive rate while focussing on the extremely low probability scenario) turn out to lead to massively incorrect result which is then counter intuitive. edit: and indeed, there are things which are correct but counter intuitive. However most of the things which are counter intuitive are also wrong; Earth being a torus is very counter intuitive and wrong, ditto for the saddle shape, etc.

Comment author: CG_Morton 12 January 2012 08:33:08PM 0 points [-]

That's hardly a critique of the trolley problem. Special relativity itself stipulates that it doesn't apply to faster-than-light movement, but a moral theory can't say "certain unlikely or confusing situations don't count". The whole point of a moral theory is to answer those cases where intuition is insufficient, the extremes you talk about. Imagine where we'd be if people just accepted Newtonian physics, saying "It works in all practical cases, so ignore the extremes at very small sizes and very high speeds, they are faulty models". Of course we don't allow that in the sciences, so why should we in ethics?

Comment author: sark 18 November 2011 10:56:26AM 5 points [-]

I'm not so sure we accord Kaj less status overall for having taking more years to graduate and more status for helping Eliezer write that book. Are we so sure we do? We might think so, and then reveal otherwise by our behavior.

In response to comment by sark on The curse of identity
Comment author: CG_Morton 18 November 2011 06:34:53PM *  6 points [-]

I can attest that I had those exact reactions on reading those sections of the article. And in general I am more impressed by someone who graduated quickly than one who took longer than average, and by someone who wrote a book rather than one who hasn't. "But what if that's not the case?" is hardly a knock-down rebuttal.

I think it's more likely you're confusing the status you attribute to Kaj for candidness and usefulness of the post, with the status you would objectively add or subtract from a person if you heard that they floundered or flourished in college.

In response to comment by [deleted] on The curse of identity
Comment author: Oscar_Cunningham 17 November 2011 02:33:26PM 4 points [-]

I want people to work toward noble efforts like charity work, but don't care much about whether they attian high status. So it's useful to aid the bit of their brain that wants to do what I want it to do.

People who care about truth might spot that part of your AI's brain wants to speak the truth, and so they will help it do this, even though this will cost it Diplomacy games. They do this because they care more about truth than Diplomacy.

Comment author: CG_Morton 18 November 2011 06:12:05PM -1 points [-]

I don't see how this is admirable at all. This is coercion.

If I work for a charitable organization, and my primary goal is to gain status and present an image as a charitable person, then efforts by you to change my mind are adversarial. Human minds are notoriously malleable, so it's likely that by insisting I do some status-less charity work you are likely to convince me on a surface level. And so I might go and do what you want, contrary to my actual goals. Thus, you have directly harmed me for the sake of your goals. In my opinion this is unacceptable.

Comment author: cata 02 November 2011 01:39:17PM *  5 points [-]

If you're in the median age band for Less Wrong, you misread the estimator. The "SAT to IQ" table is for the pre-1995 SAT, which had much more rarefied heights. The "SAT I to IQ" table is for the 1995-2005 SAT.

(I did the same thing.)

Comment author: CG_Morton 02 November 2011 02:50:17PM 1 point [-]

You are quite right. My scores correlate much better now; I retract my confusion.

Comment author: torekp 01 November 2011 01:18:27AM *  7 points [-]

Are we encouraged to estimate IQ from SAT tests and the like? That's what I did. That could reduce the excluded-middle bias that Gedusa mentions.

Comment author: CG_Morton 01 November 2011 03:29:05PM 4 points [-]

I underwent a real IQ test when I was young, and so I can say that this estimation significantly overshoots my actual score. But that's because it factors in test-taking as a skill (one that I'm good at). Then again, I'm also a little shocked that the table on that site puts an SAT score of 1420 at the 99.9th percentile. At my high school there were, to my knowledge, at least 10 people with that high of a score (and that's only those I knew of), not to mention one perfect score. This is out of ~700 people. Does that mean my school was, on average, at the 90th percentile of intelligence? Or just at the 90th percentile of studying hard (much more likely I think).

View more: Next