Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Belief in Intelligence

15 Post author: Eliezer_Yudkowsky 25 October 2008 03:00PM

Previously in seriesExpected Creative Surprises

Since I am so uncertain of Kasparov's moves, what is the empirical content of my belief that "Kasparov is a highly intelligent chess player"?  What real-world experience does my belief tell me to anticipate?  Is it a cleverly masked form of total ignorance?

To sharpen the dilemma, suppose Kasparov plays against some mere chess grandmaster Mr. G, who's not in the running for world champion.  My own ability is far too low to distinguish between these levels of chess skill.  When I try to guess Kasparov's move, or Mr. G's next move, all I can do is try to guess "the best chess move" using my own meager knowledge of chess.  Then I would produce exactly the same prediction for Kasparov's move or Mr. G's move in any particular chess position.  So what is the empirical content of my belief that "Kasparov is a better chess player than Mr. G"?

The empirical content of my belief is the testable, falsifiable prediction that the final chess position will occupy the class of chess positions that are wins for Kasparov, rather than drawn games or wins for Mr. G.  (Counting resignation as a legal move that leads to a chess position classified as a loss.)  The degree to which I think Kasparov is a "better player" is reflected in the amount of probability mass I concentrate into the "Kasparov wins" class of outcomes, versus the "drawn game" and "Mr. G wins" class of outcomes.  These classes are extremely vague in the sense that they refer to vast spaces of possible chess positions - but "Kasparov wins" is more specific than maximum entropy, because it can be definitely falsified by a vast set of chess positions.

The outcome of Kasparov's game is predictable because I know, and understand, Kasparov's goals.  Within the confines of the chess board, I know Kasparov's motivations - I know his success criterion, his utility function, his target as an optimization process.  I know where Kasparov is ultimately trying to steer the future and I anticipate he is powerful enough to get there, although I don't anticipate much about how Kasparov is going to do it.

Imagine that I'm visiting a distant city, and a local friend volunteers to drive me to the airport.  I don't know the neighborhood. Each time my friend approaches a street intersection, I don't know whether my friend will turn left, turn right, or continue straight ahead.  I can't predict my friend's move even as we approach each individual intersection - let alone, predict the whole sequence of moves in advance.

Yet I can predict the result of my friend's unpredictable actions: we will arrive at the airport.  Even if my friend's house were located elsewhere in the city, so that my friend made a completely different sequence of turns, I would just as confidently predict our arrival at the airport.  I can predict this long in advance, before I even get into the car.  My flight departs soon, and there's no time to waste; I wouldn't get into the car in the first place, if I couldn't confidently predict that the car would travel to the airport along an unpredictable pathway.

Isn't this a remarkable situation to be in, from a scientific perspective?  I can predict the outcome of a process, without being able to predict any of the intermediate steps of the process.

How is this even possible?  Ordinarily one predicts by imagining the present and then running the visualization forward in time.  If you want a precise model of the Solar System, one that takes into account planetary perturbations, you must start with a model of all major objects and run that model forward in time, step by step.

Sometimes simpler problems have a closed-form solution, where calculating the future at time T takes the same amount of work regardless of T.  A coin rests on a table, and after each minute, the coin turns over.  The coin starts out showing heads.  What face will it show a hundred minutes later?  Obviously you did not answer this question by visualizing a hundred intervening steps.  You used a closed-form solution that worked to predict the outcome, and would also work to predict any of the intervening steps.

But when my friend drives me to the airport, I can predict the outcome successfully using a strange model that won't work to predict any of the intermediate steps.  My model doesn't even require me to input the initial conditions - I don't need to know where we start out in the city!

I do need to know something about my friend.  I must know that my friend wants me to make my flight.  I must credit that my friend is a good enough planner to successfully drive me to the airport (if he wants to).  These are properties of my friend's initial state - properties which let me predict the final destination, though not any intermediate turns.

I must also credit that my friend knows enough about the city to drive successfully.  This may be regarded as a relation between my friend and the city; hence, a property of both.  But an extremely abstract property, which does not require any specific knowledge about either the city, or about my friend's knowledge about the city.

This is one way of viewing the subject matter to which I've devoted my life - these remarkable situations which place us in such an odd epistemic positions.  And my work, in a sense, can be viewed as unraveling the exact form of that strange abstract knowledge we can possess; whereby, not knowing the actions, we can justifiably know the consequence.

"Intelligence" is too narrow a term to describe these remarkable situations in full generality.  I would say rather "optimization process".  A similar situation accompanies the study of biological natural selection, for example; we can't predict the exact form of the next organism observed.

But my own specialty is the kind of optimization process called "intelligence"; and even narrower, a particular kind of intelligence called "Friendly Artificial Intelligence" - of which, I hope, I will be able to obtain especially precise abstract knowledge.

Comments (33)

Sort By: Old
Comment author: Nominull3 25 October 2008 04:05:18PM 8 points [-]

I don't know that it's that impressive. If we launch a pinball in a pinball machine, we may have a devil of a time calculating the path off all the bumpers, but we know that the pinball is going to wind up fallin in the hole in the middle. Is gravity really such a genius?

Comment author: MagnetoHydroDynamics 17 June 2012 12:19:35PM 1 point [-]

No, gravity isn't genius, because even a well-designed pinball machine can have a fault that lets the ball get stuck somewhere. You are using a closed-form solution, not an optimization process anticipation.

Gravity doesn't let water flow uphill a little in order to flow downhill a lot later.

You might not be aware of my reaction times or how good I am at pinball, but you may anticipate with high probability that if I play this pinball game, the pinball will not just wind up falling straight in the hole.

Comment author: Plasmon 17 June 2012 12:33:41PM 1 point [-]

Gravity doesn't let water flow uphill a little in order to flow downhill a lot later.

Not to detract from your main point, but superfluids do exactely that.

Comment author: MagnetoHydroDynamics 17 June 2012 12:51:51PM 1 point [-]

Superfluids still do not jump to higher energy states in order to descend to a lower one afterwards. Each atomic interaction a superfluid exhibits will always conserve energy and be, on average, entropic.

Comment author: [deleted] 04 October 2012 03:05:29PM 0 points [-]

Each move in an optimization process will increase expected utility...

Comment author: MagnetoHydroDynamics 04 October 2012 04:25:33PM 0 points [-]

Yes. The only variables are the Utility Function, the Search Space Compression and the Primitive Action Set.

Comment author: Vladimir_Nesov 25 October 2008 04:48:33PM 2 points [-]

Reminds how real "abstract" concepts are, when they get translated into simple physical events. The concept of winning, though intermediary of abstract inference, gets translated into individual physical moves, which bring the environment into a winning state. The concept of a world champion captures the whole process, and translates into an expectation of winning, maybe into an action of betting. And if you can't accurately translate these concepts into individual moves, you can't control the winning moves.

Comment author: guest4 25 October 2008 05:14:57PM 2 points [-]

"Is gravity really such a genius?"

Gravity may not be a genius, but it's still an optimization problem, since the ball "wants" to minimize its potential energy. Of course, there are limits to such a reasoning: perhaps the ball will get stuck somewhere and never reach the lowest-energy state.

Comment author: Tim_Tyler 25 October 2008 05:21:55PM -3 points [-]

Nature continually runs its own optimisation process. It maximises entropy. That's why water runs downhill, why gas expands into a vaccum, and why pinballs fall down their holes. For details, see my Bright Light essay, and the work of Roderick Dewar.

Comment author: DanielLC 05 October 2012 05:50:48AM 3 points [-]

I don't think increasing entropy can be considered an optimization process. It's not moving towards a narrow goal set. It's moving towards a wide one. What's more, the low entropy results are not disproportionately unlikely a result. The increase in entropy will not make a glass of water any less likely to spontaneously freeze than to exist in any other configuration. You're basically saying that you're 99.9999% sure that the outcome will be in a class that contains 99.9999% of all states. You know nothing.

Comment author: Emile 25 October 2008 06:02:01PM 0 points [-]

I wouldn't count gravity (or the pinball machine) as an optimization process, and I think Elizier would say the same, only better. Once you start saying that gravity, gas expansion etc. are optimization process, it kinda sounds like *everything* is an optimization process, which isn't very useful.

The pinball machine question is still usefuln because it helps refine the concept. I'd say the difference between the pinball machine and Kasparov is that once you know the probability distributions for every individual "choice" of the machine (whether a ball will bounce left or right, etc.), you know all about the machine and can use those distributions to prove the ball will reach the bottom. (If it's a bit hard to imagine for a pinball machine, so imagine a simplified model with a bunch of tubes going around, and some "random" nodes where the ball may fall left or right - like a pinball or patchinko machine, but with a countable number of paths)

Unlike the pinball machine, Having a probability distribution for each of Kasparov's choices isn't enough to predict the result of the Kasparov-Elizier game.

You need to know that Kasparov is trying to win to predict the result.

There's no equivalent knowledge for the Pinball machine - having a model of each of it's possible "choices" is enough.

Comment author: Emile 25 October 2008 06:06:40PM 0 points [-]

(damn, it's Pachinko, not patchinko)

You still need to know about gravity to make local predictions about the path of the ball in the pinball machine, but unlike for Kasparov, you don't need to think at all about future or final states.

Comment author: Emile 25 October 2008 06:28:21PM 0 points [-]

(Damn, it's Eliezer, not Elizier. Sorry!)

Comment author: Alan_Crowe 25 October 2008 06:35:00PM 0 points [-]

A similar phenomenon arises in trying to bound the error of a numerical computation by running it using interval arithmetic. The result is conservative, but sometimes useful. However, once in a while one applies it to a slowly converging iterative process that produces an accurate answer. Lots of arithmetic leads to large intervals even though the error to be bounded is small.

Comment author: RobinHanson 25 October 2008 06:41:05PM 0 points [-]

We also have concepts such as "wealth" and "power" which describe an agent's ability to achieve its goals. Will you be distinguishing "intelligence" from these, or are they synonyms for your purposes?

Comment author: DanielLC 05 October 2012 05:53:14AM 0 points [-]

"Wealth" refers to possession of valuable things specifically. "Power" seems to be more the potential to optimize rather than the act of actually optimizing.

Comment author: RobbBB 31 August 2013 11:28:41PM 0 points [-]

Wealth and intelligence are both kinds of power. The most important distinction may be that we think of wealth as outside the agent, whereas intelligence is internal to the agent. So if you change the agent's environment while leaving the agent intact, its intelligence will remain constant, but its wealth will fluctuate.

If it's hard to exactly define where an agent's boundaries are, the distinction might become merely conventional. For instance, if my normal level of intelligence is reliant on ingesting a certain food or nootropic (or meme), then that resource may be viewed as part of my intelligence (like my brain's architecture), or it may be viewed as an environmental factor that modulates how effective my intelligence is at shaping the particular environment it's in.

Comment author: james_Andrix 25 October 2008 06:50:44PM -1 points [-]

Pinball machines are optimization processes to move quarters in through a little slot and out through a little locked door.

@Nominull: Not impressive? The genius is in making a pinball machine that works. Even in the simplest case: a 'plinko' game, the board, kasparov, and the driver are all finely tuned results of some other optimization process.

Comment author: Caledonian2 25 October 2008 07:19:43PM 1 point [-]

Gravity may not be a genius, but it's still an optimization problem, since the ball "wants" to minimize its potential energy.

Using the terms as Eliezer has, can you offer an example of a phenomenon that is NOT an optimization?

Comment author: DanielLC 05 October 2012 05:56:14AM 0 points [-]

Giving an example of a phenomenon that is not an optimization is like giving an example of something without a weight. It's a sliding scale. Everything optimizes, but some things optimize better than others. A pebble doesn't optimize much of anything. An animal can optimize its inclusive genetic fitness, but not that well. Humans are better at optimizing, but don't always optimize the same thing, and often work at cross-purposes against themselves.

Comment author: RobbBB 31 August 2013 11:31:18PM 0 points [-]

Could you explain your analogy? In our universe, some things don't have mass; a sliding scale can have a 0 point, and it might be that in a certain universe almost everything falls on that 0. If your optimization power is 0 on the scale, in a sense you're an atrocious optimization process; but I think it's a bit clearer to say that you aren't an optimization process at all.

Comment author: DanielLC 01 September 2013 01:46:47AM 0 points [-]

I guess you can feasibly get zero with some things, but this is more like hitting distance zero from a bullseye. If you score 0.0001, you're an atrocious optimizer. If you're a little worse, you score -0.0001 and you're actually optimizing for the opposite effect, in which case you have score 0.0001. If you pick an entity and a goal system at random, you probably won't get a very high score for optimization, and it will be negative half the time, but it will almost never be zero. In order for an entity to not optimize any goal system, it would have to score a perfect zero for every goal system. It's not going to happen, unless you count "nothing" as your entity.

Comment author: Nornagest 01 September 2013 03:36:53AM *  0 points [-]

Wait a minute. Not everything in our universe is real-valued, much less continuous. Unless you're saying that an optimization goal must produce a well-ordering of possible environment states (which isn't true for any definition of optimization I've ever heard of in an AI context), it should be fairly easy to come up with an objective that generates a cost function returning zero for many possible hypotheses.

For example, "optimize the number of electoral votes I get in the next US presidential election".

Comment author: DanielLC 01 September 2013 03:57:56AM *  0 points [-]

Unless you're saying that an optimization goal must produce a well-ordering of possible environment states (which isn't true for any definition of optimization I've ever heard of in an AI context)

You mean an ordering? The reals aren't well-ordered.

If there's no ordering, there's circular preferences.

In any case, that's not what I was talking about.

For example, "optimize the number of electoral votes I gain in the upcoming US presidential election".

Compare the expected number of electoral votes with and without the optimizer. The difference gives you how powerful the optimizer is, and it will almost never be zero.

Comment author: Nornagest 01 September 2013 04:03:12AM *  0 points [-]

You mean an ordering? The reals aren't well-ordered.

Shoot, you're right. I believe I meant a strict ordering; it's been a while since I last studied set theory.

I'm confused as to what you mean by an optimizer now, though. It sounds like you mean something along the lines of a utility-based agent, but expected utility in this context is an attribute of a hypothesis relative to a model, not of the hypothesis relative to the world, and we're just as free to define models as we are to define optimization objectives. Previously I'd been thinking in terms of a more general agent, which needn't use a concept of utility and whose performance relative to an objective is found in retrospect.

Comment author: DanielLC 01 September 2013 06:18:24AM 1 point [-]

Previously I'd been thinking in terms of a more general agent, which needn't use a concept of utility and whose performance relative to an objective is found in retrospect.

It doesn't need to use utility explicitly. It's just whatever objective it tends to gravitate towards.

I'm not entirely sure what you're saying in the rest of the comment.

The reason I'm talking about "expected value" is that an optimizer must be able to work in a variety of environments. This is equivalent to talking about a probability distribution of environments.

Comment author: Larry_D'Anna 25 October 2008 09:17:57PM 0 points [-]

I feel like I've read this exact post before. Deja Vu?

Comment author: Tim_Tyler 25 October 2008 09:38:02PM 0 points [-]

Once you start saying that gravity, gas expansion etc. are optimization process, it kinda sounds like *everything* is an optimization process, which isn't very useful.

Entropy-generating processes are the ones that perform optimisation - specifically processes where there a range of different outcomes with different entropies. Gravity barely qualifies - but friction certainly does - and it is not gravity but friction that keeps the pinball down.

Comment author: RichardKennaway 25 October 2008 11:13:34PM 1 point [-]

And my work, in a sense, can be viewed as unraveling the exact form of that strange abstract knowledge we can possess; whereby, not knowing the actions, we can justifiably know the consequence.

You need to learn control theory. There is nothing strange about the situation you describe: this is a characteristic of all control systems, and hence of pretty much any prediction you try to make about living organisms (or AIs).

If I set the room thermostat to 20째C, I can predict what the temperature will be in the room for the indefinite future. I will not be able to predict when it will turn the heating on and off (or in hotter places, the air conditioning), because that will depend on the weather outside, which I cannot predict, and the number of people or other power sources in the room, about which I may know nothing. I can do no better than a very mushy prediction that the heating will be turned on for a greater proportion of the night than the day, and less during a LAN party. Not knowing the actions, we can justifiably know the consequence. This is an entirely unmysterious fact about control systems -- it is what they do.

It is also isomorphic to the situations of predicting that Kasparov will beat Mr. G and that the driver will arrive at the airport. People act so as to achieve goals. The actions will depend on the ongoing circumstances: each action will be whatever is effective at that moment in reaching the goal. In all three examples, the circumstances cannot be predicted, therefore the actions cannot be predicted. Not knowing the actions, we can justifiably know the consequence.

We can justifiably know the consequence because the result is not merely a consequence of the actions: the actions were chosen to produce the result. That is why we need not know of the actions. We only need know that there is a system in place that is able to choose such actions. Even when it is a black box, we can tell that there is a control system inside by observing the very fact that the result is consistent while the actions vary.

For the thermostat, we know exactly how it chooses its actions. For the driver's task, we almost know enough to duplicate the feat, if not the actual mechanism in the driver. For Kasparov, we know almost nothing about how he wins at chess. But this is merely ignorance, not mysteriousness. His track record demonstrates that he can defeat all ordinary masters of the game. Knowing that he can, we can predict that he will.

Comment author: RichardKennaway 25 October 2008 11:35:54PM 0 points [-]

Ordinarily one predicts by imagining the present and then running the visualization forward in time.

On the contrary, imagining the present and running the visualisation forwards is a very bad method of making predictions. You can only do it successfully for very simple systems, such as the planets orbiting the sun, and it doesn't work very well even for that (replacing "imagining" by "measuring" and "running the visualization forward in time" by "numerically solving Newton's laws by iterating through time"). It does not work at all for any control system, because its actions of the moment depend not only on the goal, but also on the current situation.

For example:

Last Friday there was announced a meet-up of OB people for Saturday. Several announced on OB that they would be there. I predict that the meeting took place, and that most of those who said they would be there were. If the meeting did happen, presumably everyone who went made a similar prediction, and set out expecting to meet the others.

Did anyone who made such a prediction do so by imagining the present and working forwards? I cannot see how such a thing could be done. You might not know -- I certainly do not -- where each of them would have been beforehand, what means of transportation they had available, and so on. I have no information to base such an imagining on. Those who went might be more well-informed about each other, but not to the extent of carrying out the computation. What I do know is the goals that were expressed, and on that basis alone I can predict that most of those goals were accomplished. I predict the result, while predicting none of the actions that caused the result, and imagining no trajectory linking the RSVPs to Saturday evening.

Comment author: Patrick_(orthonormal) 27 October 2008 01:06:33AM 1 point [-]

Richard, You're making the exact point Eliezer just did, about how modeling the effects of intelligence doesn't generally proceed by running a simulation forward. The "ordinarily" he speaks of, I assume, refers to the vast majority of physical systems in the Universe, in which there are no complicated optimization processes (especially intelligences) affecting outcomes on the relevant scales.

Comment author: RichardKennaway 27 October 2008 03:06:32PM 1 point [-]

Patrick(orthonormal): The "ordinarily" he speaks of, I assume, refers to the vast majority of physical systems in the Universe, in which there are no complicated optimization processes (especially intelligences) affecting outcomes on the relevant scales.

My point is that modelling the effects of unintelligence doesn't generally proceed by running a simulation forward either. No intelligence and no optimisation processes, complicated or otherwise, need be present for the system to be unpredictable by this method. The room thermostat is not intelligent. My robot is not intelligent. Neither do they optimise anything. Here is Eliezer's own example of an "ordinary" system:

Ordinarily one predicts by imagining the present and then running the visualization forward in time. If you want a precise model of the Solar System, one that takes into account planetary perturbations, you must start with a model of all major objects and run that model forward in time, step by step.

But this is, in fact, not how astronomers precisely predict the future positions of the bodies of the Solar System. They do not "run a model forward in time, step by step". Instead, from observations they compute a set of parameters ("orbital elements") of the closed-form solution to the two-body problem (the second body being the Sun), then add perturbative adjustments to account for interactions between bodies. This is a more accurate method, at least when the bodies do not perturb each other too much. It also allows future positions to be predicted without computing any of the intermediate positions -- without, that is, running a model forwards in time.

So not only does forward simulation not work at all for intelligent systems, neither does it work at all for unintelligent control systems, and it does not even work very well for a bunch of dumb rocks.

Both forward simulation and the method of elements and perturbations are mathematically derived from Newton's laws. How is it that the latter method can predict where an asteroid will be at a point in the future, not only without computing its entire trajectory, but more accurately than if we did? It is because Newton's laws tell us more than the moment-to-moment evolution. They mathematically imply long-range properties of that evolution that allow these predictions to be made.

Comment author: BaconServ 08 May 2012 05:23:25AM 0 points [-]

<author>

Oh the comments here are a sad state of affairs. :(

And this is an excessively important article too.