shminux comments on Policy Debates Should Not Appear One-Sided - Less Wrong

102 Post author: Eliezer_Yudkowsky 03 March 2007 06:53PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (179)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: DaFranker 18 January 2013 07:43:57PM *  1 point [-]

Soooo...

Suppose I hypnotize all humans. All of them! And I give them all the inviolable command to always praise murder and genocide. I'm so good at hypnosis that it overrides everything else and this Law becomes a tightly-entangled part of their entire consciousnesses. However, they still hate murder and genocide, are still unhappy about their effects, etc. They just praise it, both vocally and internally and mentally. Somewhat like how many used to praise Zeus, despite most of his interactions with the world being "Rape people" and "Kill people".

By the argument you're giving, this would effectively hack and reprogram morality itself (gasp!) such that you should always do murder and genocide as much as possible (since they "always" praise it, without diminishing returns or habituation effects or desensitization).

Clearly this is not the same as what you ought to do.

(In this case, my first guess would be that you should revert my hypnosis and prevent me and anyone else from ever doing that again.)

For more exploration into this, suppose I'm always optimally good. Always. A perfectly optimally-morally-good human. What praise do I get? Well, some for that, some once in a while when I do something particularly heroic. Otherwise, various effects make the praise rather rare.

On the other hand, if I'm a super-sucky bad human that kills people by accident all the time (say, ten every hour on average), then each time I manage to prevent one such accident I get praise. I could optimize this and generate a much larger amount of praise with this strategy. Clearly this set of action attracts more praise. Should I ought to do this and seek to do it more than the previous one?

Comment author: shminux 18 January 2013 08:25:17PM 0 points [-]

However, they still hate murder and genocide, are still unhappy about their effects, etc. They just praise it, both vocally and internally and mentally.

How can you hate something yet praise it internally? I'm having trouble coming up with an example.

Comment author: DaFranker 18 January 2013 08:30:25PM 4 points [-]

I know a very good one, very grounded in reality, that millions if not billions of people have and do this.

Death.