Nebu comments on The Blue-Minimizing Robot - Less Wrong

162 Post author: Yvain 04 July 2011 10:26PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (159)
Tags: motivation

Comments (159)

You are viewing a single comment's thread. Show more comments above.

Comment author: JamesAndrix 06 July 2011 10:04:35PM 1 point [-]

It seems that the narrative of unfriendly AI is only a risk if an AI were to have a true goal function, and many useful advances in artificial intelligence (defined in the broad sense) carry no risk of this kind.

What does it mean for a program to have intelligence if it does not have a goal? (or have components that have goals)

The point of any incremental intelligence increase is to let the program make more choices, and perhaps choices at higher levels of abstraction. Even at low intelligence levels, the AI will only 'do a good job' if the basis of those choices adequately matches the basis we would use to make the same choice. (a close match at some level of abstraction below the choice, not the substrate and not basic algorithms)

Creating 'goal-less' AI still has the machine making more choices for more complex reasons, and allows for non-obvious mismatches between what it does and what we intended it to do.

Yes, you can look at paperclip-manufacturing software and see that it is not a paper-clipper, but some component might still be optimizing for something else entirely. We can reject the anthropomorphically obvious goal and there can still be an powerful optimization process that affects the total system, at the expense of both human values and produced paperclips.

Comment author: Nebu 11 December 2015 05:31:04AM 0 points [-]

What does it mean for a program to have intelligence if it does not have a goal?

This is a very interesting question, thanks for making me think about it.

(Based on your other comments elsewhere in this thread), it seems like you and I are in agreement that intelligence is about having the capability to make better choices. That is, two agents given an identical problem and identical resources to work with, the agent that is more intelligent is more likely to make the "better" choice.

What does "better" mean here? We need to define some sort of goal and then compare the outcome of their choices and how closely those outcome matches those goals. I have a couple of disorganized thoughts here:

  • The goal is just necessary for us, outsiders, to compare the intelligence of the two agents. The goal is not necessary for the existence of intelligence in the agents if no one's interested in measuring their intelligence.
  • Assuming the agents are cooperative, you can temporarily assign subgoals. For example, perhaps you and I would like to know which one of us is smarter. You and I might have many different goals, but we might agree to temporarily take on a similar goal (e.g. win this game of chess, or get the highest amount of correct answers on this IQ test, etc.) so that our intelligence can be compared.
  • The "assigning" of goals to an intelligence strongly implies to me that goals are orthogonal to intelligence. Intelligence is the capability to fulfil any general goal, and it's possible for someone to be intelligent even if they do not (currently, or ever) have any goals. If we come up with a new trait called Sodadrinkability which is the capability to drink a given soda, one can say that I possess Sodadrinkability -- that I am capable of drinking a wide range of possible sodas provided to me -- even if I do not currently (or ever) have any sodas to drink.
Powered by Reddit Powered by Reddit