Previously in series: Measuring Optimization Power
Is Deep Blue "intelligent"? It was powerful enough at optimizing chess boards to defeat Kasparov, perhaps the most skilled chess player humanity has ever fielded.
A bee builds hives, and a beaver builds dams; but a bee doesn't build dams and a beaver doesn't build hives. A human, watching, thinks, "Oh, I see how to do it" and goes on to build a dam using a honeycomb structure for extra strength.
Deep Blue, like the bee and the beaver, never ventured outside the narrow domain that it itself was optimized over.
There are no-free-lunch theorems showing that you can't have a truly general intelligence that optimizes in all possible universes (the vast majority of which are maximum-entropy heat baths). And even practically speaking, human beings are better at throwing spears than, say, writing computer programs.
But humans are much more cross-domain than bees, beavers, or Deep Blue. We might even conceivably be able to comprehend the halting behavior of every Turing machine up to 10 states, though I doubt it goes much higher than that.
Every mind operates in some domain, but the domain that humans operate in isn't "the savanna" but something more like "not too complicated processes in low-entropy lawful universes". We learn whole new domains by observation, in the same way that a beaver might learn to chew a different kind of wood. If I could write out your prior, I could describe more exactly the universes in which you operate.
Is evolution intelligent? It operates across domains - not quite as well as humans do, but with the same general ability to do consequentialist optimization on causal sequences that wend through widely different domains. It built the bee. It built the beaver.
Whatever begins with genes, and impacts inclusive genetic fitness, through any chain of cause and effect in any domain, is subject to evolutionary optimization. That much is true.
But evolution only achieves this by running millions of actual experiments in which the causal chains are actually played out. This is incredibly inefficient. Cynthia Kenyon said, "One grad student can do things in an hour that evolution could not do in a billion years." This is not because the grad student does quadrillions of detailed thought experiments in their imagination, but because the grad student abstracts over the search space.
By human standards, evolution is unbelievably stupid. It is the degenerate case of design with intelligence equal to zero, as befitting the accidentally occurring optimization process that got the whole thing started in the first place.
(As for saying that "evolution built humans, therefore it is efficient", this is, firstly, a sophomoric objection; second, it confuses levels. Deep Blue's programmers were not superhuman chessplayers. The importance of distinguishing levels can be seen from the point that humans are efficiently optimizing human goals, which are not the same as evolution's goal of inclusive genetic fitness. Evolution, in producing humans, may have entirely doomed DNA.)
I once heard a senior mainstream AI type suggest that we might try to quantify the intelligence of an AI system in terms of its RAM, processing power, and sensory input bandwidth. This at once reminded me of a quote from Dijkstra: "If we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger." If you want to measure the intelligence of a system, I would suggest measuring its optimization power as before, but then dividing by the resources used. Or you might measure the degree of prior cognitive optimization required to achieve the same result using equal or fewer resources. Intelligence, in other words, is efficient optimization.
So if we say "efficient cross-domain optimization" - is that necessary and sufficient to convey the wisest meaning of "intelligence", after making a proper effort to factor out anthropomorphism in ranking solutions?
I do hereby propose: "Yes."
Years ago when I was on a panel with Jaron Lanier, he had offered some elaborate argument that no machine could be intelligent, because it was just a machine and to call it "intelligent" was therefore bad poetry, or something along those lines. Fed up, I finally snapped: "Do you mean to say that if I write a computer program and that computer program rewrites itself and rewrites itself and builds its own nanotechnology and zips off to Alpha Centauri and builds its own Dyson Sphere, that computer program is not intelligent?"
This, I think, is a core meaning of "intelligence" that it is wise to keep in mind.
I mean, maybe not that exact test. And it wouldn't be wise to bow too directly to human notions of "impressiveness", because this is what causes people to conclude that a butterfly must have been intelligently designed (they don't see the vast incredibly wasteful trail of trial and error), or that an expected paperclip maximizer is stupid.
But still, intelligences ought to be able to do cool stuff, in a reasonable amount of time using reasonable resources, even if we throw things at them that they haven't seen before, or change the rules of the game (domain) a little. It is my contention that this is what's captured by the notion of "efficient cross-domain optimization".
Occasionally I hear someone say something along the lines of, "No matter how smart you are, a tiger can still eat you." Sure, if you get stripped naked and thrown into a pit with no chance to prepare and no prior training, you may be in trouble. And by similar token, a human can be killed by a large rock dropping on their head. It doesn't mean a big rock is more powerful than a human.
A large asteroid, falling on Earth, would make an impressive bang. But if we spot the asteroid, we can try to deflect it through any number of methods. With enough lead time, a can of black paint will do as well as a nuclear weapon. And the asteroid itself won't oppose us on our own level - won't try to think of a counterplan. It won't send out interceptors to block the nuclear weapon. It won't try to paint the opposite side of itself with more black paint, to keep its current trajectory. And if we stop that asteroid, the asteroid belt won't send another planet-killer in its place.
We might have to do some work to steer the future out of the unpleasant region it will go to if we do nothing, but the asteroid itself isn't steering the future in any meaningful sense. It's as simple as water flowing downhill, and if we nudge the asteroid off the path, it won't nudge itself back.
The tiger isn't quite like this. If you try to run, it will follow you. If you dodge, it will follow you. If you try to hide, it will spot you. If you climb a tree, it will wait beneath.
But if you come back with an armored tank - or maybe just a hunk of poisoned meat - the tiger is out of luck. You threw something at it that wasn't in the domain it was designed to learn about. The tiger can't do cross-domain optimization, so all you need to do is give it a little cross-domain nudge and it will spin off its course like a painted asteroid.
Steering the future, not energy or mass, not food or bullets, is the raw currency of conflict and cooperation among agents. Kasparov competed against Deep Blue to steer the chessboard into a region where he won - knights and bishops were only his pawns. And if Kasparov had been allowed to use any means to win against Deep Blue, rather than being artificially restricted, it would have been a trivial matter to kick the computer off the table - a rather light optimization pressure by comparison with Deep Blue's examining hundreds of millions of moves per second, or by comparison with Kasparov's pattern-recognition of the board; but it would have crossed domains into a causal chain that Deep Blue couldn't model and couldn't optimize and couldn't resist. One bit of optimization pressure is enough to flip a switch that a narrower opponent can't switch back.
A superior general can win with fewer troops, and superior technology can win with a handful of troops. But even a suitcase nuke requires at least a few kilograms of matter. If two intelligences of the same level compete with different resources, the battle will usually go to the wealthier.
The same is true, on a deeper level, of efficient designs using different amounts of computing power. Human beings, five hundred years after the Scientific Revolution, are only just starting to match their wits against the billion-year heritage of biology. We're vastly faster, it has a vastly longer lead time; after five hundred years and a billion years respectively, the two powers are starting to balance.
But as a measure of intelligence, I think it is better to speak of how well you can use your resources - if we want to talk about raw impact, then we can speak of optimization power directly.
So again I claim that this - computationally-frugal cross-domain future-steering - is the necessary and sufficient meaning that the wise should attach to the word, "intelligence".
I once read an interesting book by Kurt Vonnegut called Player Piano that focused on an increasingly automated society (where one was not allowed to have a 'creative' job, for example, unless the punch card that assessed your general intelligence by way of standardized test happened to have the 'creative' slot punched; and where out of work (displaced by robots) auto mechanics pined over the good ole days when they could ply their trade for a living rather than seeing their vocation-which-they-also-had-creative-passion-for become the work of robots.)
In particular, a character (a discontented engineer who goes Luddite and becomes a farmer) has this to say in a climactic letter in the story:
“You perhaps disagree with the antique and vain notion of Man’s being a creation of God. ... But I find it a far more defensible belief than the one implicit in intemperate faith in lawless technological progress—namely, that man is on earth to create more durable and efficient images of himself, and, hence, to eliminate any justification at all for his own continued existence.”
I found this quite interesting and simply wonder what LWers think about this premise. I disagree with the idea that religious faith in God is somehow more defensible than persistent technological progress for the sake of making myself more durable and efficient (and presumably we can extend the quote to include 'rational'). However, I don't necessarily disagree outright that "intemperate faith in lawless technological progress—namely, that man is on earth to create more durable and efficient images of himself, and, hence, to eliminate any justification at all for his own continued existence" -- is not strongly defensible. That phrase at the end ... "eliminate any justification for his own continued existence" is quite resonant.
Might we not be able to define intelligence in this way? Intelligence is a property such that once I have it, then I am personally less on the hook for expending my own resources to sustain my own existence. In a slight sense, intelligence renders its owner less and less necessary for the actual work that goes into self preservation.
The sense in which I think this applies is as follows: suppose my utility function heavily values my perception of sustained existence. But at the same time, at least some component of my utility is derived from perceiving a "sense of meaning" out of life (a concrete definition of that could be debated endlessly, so please lets just go with a basic natural language understanding of that for right now). If I feel that increase in intelligence improves probability of survival but diminishes personal usefulness and "meaning", then it is at least possible that there is some finite largest amount of intelligence (quantified in terms of efficient optimization power if you so choose) such that beyond that intelligence horizon, my perceived quality of life actually decreases because my perceived personal meaning drops low enough to offset the marginal increased assurance of longer(/more comfortable) life.
I think this is a non-trivial theory about intelligence. This "intelligence horizon" may not be something that humans could even begin to encounter, as most of us seem perfectly capable of extracting "meaning" out of life despite technological progress. But this isn't the same as knowing a fundamental reason why all intelligences would always value a unit improvement in longevity over a unit improvement in "personal meaning".
For example, suppose that many billions of years into the future there have taken place several great battles between Bayesian superintelligences forced to hostility over scarcity of resource so severe that even their combined superintelligent efforts at collaboration could not solve the problems. One triumphant superintelligence remains and is confident that the probability it is the last remaining life within its light cone is near 1. Suppose it goes on to self colonize territory and resources until it is confident that the probability that there is additional resource to consume is near zero (here I mean that it believes with a high degree of confidence that it has located all available resource that it can physically access).
Now what does it do? What is the psychology of such a being? I don't even pretend to know a good answer to this sort of question, but I am sure other LWers will have good things to say. But you can clearly see that as this last Bayesian intelligence completes tasks, it directly loses purpose. Unless it does not weigh a sense of purpose into its utility function, this would be problematic. Maybe it would start to intentionally solve problems very slowly so that it required a very long time to finish. Maybe its slowness would increase as time goes on to ensure that it never actually does finish the task?
It'd be cool if, based on knowing something about your light cone, the problems needing solved during your lifetime, and the imbued method by which your utility function assigns weight to a sense of purpose, you could compute an optimum life span... I mean, suppose the universe was the interval [1,N] for some very large N and my utility function under the scenario where I set myself to just optimize indefinitely and make myself more durable and efficient starts looking like 1/x^2 after a long time, say from integer M to N where M is also very large. In order that we can even have an expected utility (which I think is a reasonable assumption) then any utility I choose ought to be rectifiable on [1,N], so no matter what the integral works out to be, I could find utility functions that actually hit zero (assuming non-negative utility, or hitting -max{utility} for bounded but possibly negative utility, or the debate is over if death -> -\infty utility but this doesn't seem plausible given the occurrence of suicide) at a time before N (corresponding to death before N) but for which the total experienced utility is higher.
[Note: this is not an attempt to pin down every rigorous detail about utility functions... just to illustrate the naive, simplistic view that even simple utility scenarios can lead to counter-intuitive decisions. The opportunity to have these sorts of situations would only increase as utility functions become more realistic and are based upon more realistic models of the universe. Imagine if you had access to your own utility function and a Taylor series approximation of the evolving quantum states that will "affect you". If you can compute any kind of meaningful cohesive extrapolated volition, why couldn't you predict your reaction to the predicament of being in a situation like that of Sisyphus and whether or not dying earlier would be better if the overall effect was that your total experienced utility increased? Can there possibly be experiences of finite duration that are so awesome in terms of utility that they outweigh futures in which you are dead? What if you were a chess player and your chess utility function was such that if you could promote 6 pawns to queens that your enjoyment of that bizarre novelty occurring during gameplay was worth way more than winning the game and so even if you saw that the opponent had a forced mate, you'd willingly walk right into it in order to get the sweet sweet novelty of the 6 queens?]
A simple example would be that, when facing the choice between unimaginable torture from which you willingly accept that the probability of escape is near zero or suicide, there could be physically meaningful utility functions (perhaps even 'optimal' utility functions) that would rationally choose suicide. A Bayesian superintelligence might view a solitary light cone existence with no tasks to derive purpose from as equivalent to a form of this sort of torture and thus see that if it just "acts dumber but dies sooner but lives the remaining years with more enjoyable purpose" it will actually have more total utility. Maybe it will shoot itself in its superintelligent foot and erase all memory of the shooting and replace it with some different explanation for why there is a bullet hole. Then the future stupider AI will be in a predicament where it realizes it needs to make itself smarter (up to the former level of intelligence that it erased from its own memory) but is somehow handicapped in a way that it's just barely out of reach (but chasing the ever-out-of-reach carrot gives it meaning). Would it rather be Sisyphus or David Foster Wallace or the third option that I am most likely failing to see?
Addition
I thought of one idea for what the Bayesian intelligence might do after all goals had been achieved for its own self-preservation, as near as it could reckon. The idea is that is might run a simulation resulting in other lifeforms. Would it be fair to say that the probability of it choosing to do this skyrockets to close to 1 as the total number of other living beings goes toward zero? If so, then as per the link above, does this make us more compelled to believe we're currently in the simulation of a last Bayesian intelligence in a significantly mature metaverse (don't tell the Christians)? My gut says probably not but I can't think of concrete compelling reasons for why this can be easily dismissed. I'll feel better if anyone can Swiss-cheesify my line of thinking.
This should be a post, on a personal blog if not on LW.