Comment author:[deleted]
06 July 2011 04:04:01PM
*
28 points
[-]
The conclusion I'd draw from this essay is that one can't necessarily derive a "goal" or a "utility function" from all possible behavior patterns. If you ask "What is the robot's goal?", the answer is, "it doesn't have one," because it doesn't assign a total preference ordering to states of the world. At best, you could say that it prefers state [I SEE BLUE AND I SHOOT] to state [I SEE BLUE AND I DON'T SHOOT]. But that's all.
This has some implications for AI, I think. First of all, not every computer program has a goal or a utility function. There is no danger that your TurboTax software will take over the world and destroy all human life, because it doesn't have a general goal to maximize the number of completed tax forms. Even rather sophisticated algorithms can completely lack goals of this kind -- they aren't designed to maximize some variable over all possible states of the universe. It seems that the narrative of unfriendly AI is only a risk if an AI were to have a true goal function, and many useful advances in artificial intelligence (defined in the broad sense) carry no risk of this kind.
Do humans have goals? I don't know; it's plausible that we have goals that are complex and hard to define succinctly, and it's also plausible that we don't have goals at all, just sets of instructions like "SHOOT AT BLUE." The test would seem to be if a human goal of "PROMOTE VALUE X" continues to imply behaviors in strange and unfamiliar circumstances, or if we only have rules of behavior in a few common situations. If you can think clearly about ethics (or preferences) in the far future, or the distant past, or regarding unfamiliar kinds of beings, and your opinions have some consistency, then maybe those ethical beliefs or preferences are goals. But probably many kinds of human behavior are more like sets of instructions than goals.
Comment author:dfpolis
15 April 2015 04:11:08PM
*
0 points
[-]
Let me suggest that the difference between goal-less behavior and goal-driven behavior, is that goal-driven behavior seeks means to attain its end. The means will vary with circumstances, while the end remains relatively invariant. Another indication of goal-driven behavior is that means are often prepared in anticipation of need, rather than in response to present need.
I said "relatively invariant" because goals can be and often are heirarical. An example was outlined by Maslow in his "A Theory of Human Motivation" in the Psychological Review (1943). Maslow aside, in problem solving, we often resort to staged solutions in which the means to a higher order goal become a new sub-goal and so on iteratively -- until we reach low level goals within our immediate grasp.
A second point is that terms such as "purposeful" and "goal-seeking" are analogously predicated. To be analogousley predicated, a term is applied to differnt cases with a meaning that is partly the same and partly different. Thus, a goal-seeking robot is not goal-seeking because it intends any goals of its own, but because it is the vehicle by which designer seeks to effect his or her goals. In the parable, if the goal was the destruction of blue uniformed enemies, that goal was only intended by the robots creators. Since the robot is an instantiated means of attaining that gosl, we may speak, analogously, of it as having the same goal. The important point is that we mean different things in saying "the designer has a goal" and "the robot has a goal." Each works toward same end (so the meaning iis partly the same), but only the designer intends that end (so the meaning iis partly different). (BTW this kind of analogy is an "analogy of attribution.")
The fact that the robot is ineffective in attaining its end is a side issue that might be solved by employing better algoritms (edge and pattern recognition, etc.) There is no evidence that better algorithms will give the robot intentions in the sense that the designer has intentions.
= 27d567bc2d48d9f40e9ccc20ea370b15)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
The conclusion I'd draw from this essay is that one can't necessarily derive a "goal" or a "utility function" from all possible behavior patterns. If you ask "What is the robot's goal?", the answer is, "it doesn't have one," because it doesn't assign a total preference ordering to states of the world. At best, you could say that it prefers state [I SEE BLUE AND I SHOOT] to state [I SEE BLUE AND I DON'T SHOOT]. But that's all.
This has some implications for AI, I think. First of all, not every computer program has a goal or a utility function. There is no danger that your TurboTax software will take over the world and destroy all human life, because it doesn't have a general goal to maximize the number of completed tax forms. Even rather sophisticated algorithms can completely lack goals of this kind -- they aren't designed to maximize some variable over all possible states of the universe. It seems that the narrative of unfriendly AI is only a risk if an AI were to have a true goal function, and many useful advances in artificial intelligence (defined in the broad sense) carry no risk of this kind.
Do humans have goals? I don't know; it's plausible that we have goals that are complex and hard to define succinctly, and it's also plausible that we don't have goals at all, just sets of instructions like "SHOOT AT BLUE." The test would seem to be if a human goal of "PROMOTE VALUE X" continues to imply behaviors in strange and unfamiliar circumstances, or if we only have rules of behavior in a few common situations. If you can think clearly about ethics (or preferences) in the far future, or the distant past, or regarding unfamiliar kinds of beings, and your opinions have some consistency, then maybe those ethical beliefs or preferences are goals. But probably many kinds of human behavior are more like sets of instructions than goals.
Let me suggest that the difference between goal-less behavior and goal-driven behavior, is that goal-driven behavior seeks means to attain its end. The means will vary with circumstances, while the end remains relatively invariant. Another indication of goal-driven behavior is that means are often prepared in anticipation of need, rather than in response to present need.
I said "relatively invariant" because goals can be and often are heirarical. An example was outlined by Maslow in his "A Theory of Human Motivation" in the Psychological Review (1943). Maslow aside, in problem solving, we often resort to staged solutions in which the means to a higher order goal become a new sub-goal and so on iteratively -- until we reach low level goals within our immediate grasp.
A second point is that terms such as "purposeful" and "goal-seeking" are analogously predicated. To be analogousley predicated, a term is applied to differnt cases with a meaning that is partly the same and partly different. Thus, a goal-seeking robot is not goal-seeking because it intends any goals of its own, but because it is the vehicle by which designer seeks to effect his or her goals. In the parable, if the goal was the destruction of blue uniformed enemies, that goal was only intended by the robots creators. Since the robot is an instantiated means of attaining that gosl, we may speak, analogously, of it as having the same goal. The important point is that we mean different things in saying "the designer has a goal" and "the robot has a goal." Each works toward same end (so the meaning iis partly the same), but only the designer intends that end (so the meaning iis partly different). (BTW this kind of analogy is an "analogy of attribution.")
The fact that the robot is ineffective in attaining its end is a side issue that might be solved by employing better algoritms (edge and pattern recognition, etc.) There is no evidence that better algorithms will give the robot intentions in the sense that the designer has intentions.