Grognor comments on SotW: Be Specific - Less Wrong

37 Post author: Eliezer_Yudkowsky 03 April 2012 06:11AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (306)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 06 April 2012 02:13:38AM *  2 points [-]

Since it's malicious, it will probably search through all of your desires, pick one of them at random to count as "my utility function", and then reinterpret the body of the wish to maximise that one thing at the expense of all others.

It's malicious and omnipotent. It'll do far worse than that. It'll scan your preferences until it finds a contradiction. Once you have a contradiction you can derive absolutely anything. It would then proceed to calculate your Coherent Extrapolated Volition and minimise it. It may not be obliged to figure out what you actually want but it can certainly do so for the purpose of being spiteful!

Comment author: Grognor 07 April 2012 07:17:35PM 6 points [-]

I think that is the first time I've ever seen anyone accurately describe the worst thing that could possibly happen.