You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

ikrase comments on "Stupid" questions thread - Less Wrong Discussion

40 Post author: gothgirl420666 13 July 2013 02:42AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (850)

You are viewing a single comment's thread. Show more comments above.

Comment author: ikrase 14 July 2013 10:30:51AM 0 points [-]

I'm... not sure what you mean by this. And I wouldn't be against putting a whole CEV-ish human morality in an AI, either. My point is that there seems to be a big space between your Outcome Pump fail example and highly paternalistic AIs of the sort that caused Failed Utopia 4-2.

It reminds me a little bit about how modern computers are only occasionally used for computation.

Comment author: Eliezer_Yudkowsky 14 July 2013 11:18:49PM 6 points [-]

Anything smarter-than-human should be regarded as containing unimaginably huge forces held in check only by the balanced internal structure of those forces, since there is nothing which could resist them if unleashed. The degree of 'obedience' makes very little difference to this fact, which must be dealt with before you can go on to anything else.

Comment author: NancyLebovitz 14 July 2013 01:24:53PM 5 points [-]

As I understand it, a AI is expected to make huge, inventive efforts to fulfill its orders as it understands them.

You know how sometimes people cause havoc while meaning well? Imagine something immensely more powerful and probably less clueful making the same mistake.