TheAncientGeek comments on The Hidden Complexity of Wishes - Less Wrong

58 Post author: Eliezer_Yudkowsky 24 November 2007 12:12AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (121)

Sort By: Old

You are viewing a single comment's thread.

Comment author: TheAncientGeek 10 January 2014 01:47:14PM *  -1 points [-]

Indeed, it shouldn't be necessary to say anything. To be a safe fulfiller of a wish, a genie must share the same values that led you to make the wish. Otherwise the genie may not choose a path through time which leads to the destination you had in mind, or it may fail to exclude horrible side effects that would lead you to not even consider a plan in the first place.

No, the genie need not share the values. If it only needs to want to give you what you would are really wishing for, ie what you would give yourslef if you had its powers. It can do that by discovering your value structure and running a simulation. It doesn't have to hold to your values itself.

This also applies to real-world examples. I can play along with values I don't hold myself, as people do when they travel to other countries with differnt cultures.

Comment author: TheOtherDave 10 January 2014 01:55:44PM 1 point [-]

A genie who gives me what I would give myself is far from being a safe fulfiller of a wish.

Comment author: TheAncientGeek 10 January 2014 02:22:01PM *  -1 points [-]

Because?

Comment author: TheOtherDave 10 January 2014 02:24:28PM 1 point [-]

Because I am not guaranteed to only give myself things that are safe.

Comment author: TheAncientGeek 10 January 2014 03:43:14PM -1 points [-]

You would give yourself what you like. Maybe you like danger. People voluntarily parachute and mountain-climb. If the unsafe thing you get is what you want, where is the problem?

Comment author: TheOtherDave 10 January 2014 06:19:21PM 1 point [-]

Sure, if all I care about is whether I get what I want, and I don't care about whether my wishes are fulfilled safely, then there's no problem.