You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Dmytry comments on [draft] Concepts are Difficult, and Unfriendliness is the Default: A Scary Idea Summary - Less Wrong Discussion

7 Post author: Kaj_Sotala 31 March 2012 10:07AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

You are viewing a single comment's thread. Show more comments above.

Comment author: Dmytry 01 April 2012 09:18:19AM *  -1 points [-]

How do you think the "Greenpeace by default" AI might define either "harm" or "value", and "life"?

It simply won't. Harm, value, life, we never defined those; they are the commonly agreed upon labels which we apply to things for communication purposes, and it works on a limited set of things that already exist but does not define anything outside context of this limited set.

It would have maximization of some sort of complexity metric (perhaps while acting conservatively and penalizing actions it can't undo to avoid self harm in the form of cornering oneself), which it first uses on itself to self improve for a while without even defining what self is. Consider evolution as example; it doesn't really define fitness in the way that humans do. It doesn't work like - okay we'll maximize the fitness that is defined so and so, so there's what we should do.

edit: that is to say, it doesn't define 'life' or 'harm'. It has a simple goal system involving some metrics, which incidentally prevents the self harm, and permits self improvement, in the sense that we would describe it this way like we would describe the shooting-at-short-part-of-visible-spectrum robot as blue-minimizing one (albeit that is not very good analogy as we define blue and minimization independently of the robot).