MugaSofer comments on By Which It May Be Judged - LessWrong

35 Post author: Eliezer_Yudkowsky 10 December 2012 04:26AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (934)

You are viewing a single comment's thread. Show more comments above.

Comment author: MugaSofer 26 December 2012 05:48:46PM 0 points [-]

I'm sorry, I don't understand what you mean when you say 'fighting the counterfactual'.

Try the search bar. It's a pretty common concept here, although I don't recall where it originated.

I was just talking about sets of desires that clash in principle. When you have to desires that clash over one thing, then you will act to fulfill the stronger of your desires. But, as I've tried to make clear, if one desire is to 'kill all humans' and another is 'to save all humans' then the best idea is to (attempt to) self-modify to have only the desire that will produce the most utility. Having both will mean disutility always.

Well, that disutility is only lower according to my new preferences; my old one's remain sadly unfulfilled.

More specifically, if I value both freedom and safety (for everyone), should I self-modify not to hate reprogramming others? Or not to care that people will decide to kill each other sometimes?

Comment author: Benito 26 December 2012 07:30:34PM 1 point [-]

Hmm... I don't think my point necessarily helps here. I meant that you will always get disutility when you have two desires that always clash (x and not x); whichever way you choose, the other desire won't be fulfilled.

However, in the case you offered (and probably most cases) it's not a good idea to self-modify, as desires don't clash in principle, always. Like with the chocolate and saving kids one, you just have to perform utility calculations to see which way to go (that one is saving kids).

Comment author: MugaSofer 27 December 2012 02:28:10AM 0 points [-]

you will always get disutility when you have two desires that always clash (x and not x); whichever way you choose, the other desire won't be fulfilled.

Yup. And if you stop caring about one of those values, then modified!you will be happier. But you don't care about what modified!you wants, you care about x and not-x.