TheAncientGeek comments on The Hidden Complexity of Wishes - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (121)
No, the genie need not share the values. If it only needs to want to give you what you would are really wishing for, ie what you would give yourslef if you had its powers. It can do that by discovering your value structure and running a simulation. It doesn't have to hold to your values itself.
This also applies to real-world examples. I can play along with values I don't hold myself, as people do when they travel to other countries with differnt cultures.
A genie who gives me what I would give myself is far from being a safe fulfiller of a wish.
Because?
Because I am not guaranteed to only give myself things that are safe.
You would give yourself what you like. Maybe you like danger. People voluntarily parachute and mountain-climb. If the unsafe thing you get is what you want, where is the problem?
Sure, if all I care about is whether I get what I want, and I don't care about whether my wishes are fulfilled safely, then there's no problem.