Vladimir_Nesov comments on How Not to be Stupid: Adorable Maybes - Less Wrong

-2 Post author: Psy-Kosh 29 April 2009 07:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (54)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 30 April 2009 12:30:05AM *  1 point [-]

That is a right sentiment about strength: there are no simple rules, only goals, which makes a creative mind extremely dangerous. And we shouldn't build things like this without understanding what the outcome will be. This is one of the reasons it's important to understand human values in this light, to guard them from this destructive potential.

Whatever you want accomplished, whatever you want averted, instrumental rationality defines an optimal way of doing that (without necessarily giving the real-world means, that's a next step). If you really want life to continue as before, the correctly implemented explicit utility function for doing that won't lead a Bayesian optimizer to do something horrible. (Although inaction may be considered horrible in itself, where so much more could've been done.)