Vladimir_Nesov comments on How Not to be Stupid: Adorable Maybes - Less Wrong

-2 Post author: Psy-Kosh 29 April 2009 07:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (54)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 30 April 2009 12:15:21AM *  3 points [-]

But that's not a hit against decision theory. That's a hit against bad utility functions.

We know from Eliezer's writings that almost any strong goal-directed chessplayer AI will destroy the world. Well guess what, if a non-world-destroying utility function appears almost impossibly hard to formulate, in my book it counts as a hit against the concept of utility functions. Especially seeing as machines based on e.g. control theory (RichardKennaway) behave much more sensibly - they almost never display any urge to screw up the whole world, instead being content to sit there and tweak their needle.

Comment author: Vladimir_Nesov 30 April 2009 12:38:24AM 0 points [-]

I agree. But what do you do with this situation? To give up, you have to be certain that there is no way out, and we are much too confused to say anything like that yet. Someone is bound to build a doom machine someday if you don't do something about it.