To describe the universe well, you will have to distinguish these signatures from each other, and have separate names for "human intelligence", "evolution", "proteins", and "protons", because even if these things are related they are not at all the same.

Speaking of separate names, I think you shouldn't call this "steering the future" stuff "intelligence." It sounds very useful, but almost no one except you is referring to it when they say "intelligence." There's some overlap, and you may feel that what you are referring to is more useful than what they are referring to, but that doesn't give you license to take over the word.

I know you've written a bunch of articles justifying your definition. I read them. I also read the articles moaning that no one understands what you mean when you say "intelligence." I think it's because they use that word to mean something else. So maybe you should just use a different word.

In fact, I think you should look at generalized optimization as a mode of analysis, rather than a (non-fundamental) property. Say, "Let's analyze this in terms of optimization (rather than conserved quantities, economic cost/benefit, etc.)" not, "Let's measure its intelligence."

In one of your earlier posts, people were saying that your definitions are too broad and therefore useless. I agree with them about the broadness, but I think this is still a useful concept if it is taken to be a one of many ways of looking at almost anything.

There are those who will see it as almost a religious principle that no one can possibly know that a design will work, no matter how good the argument, until it is actually tested.

It's not necessarily a religious principle, although like anything else, it can be made one. It's a simple truth.

There is a non-trivial distinction between believing that a design will work, and believing that a design is highly likely to work. Maintaining the awareness that our maps do not constrain the territory is hard work, but necessary for a rationalist.

Analogously, we can see that nearly all of the optimization power of human engineering must have already been exerted in coming up with good designs to test. The final selection on the basis of good results is only the icing on the cake. If you test four designs that seem like good ideas, and one of them works best, then at most 2 bits of optimization pressure can come from testing - the rest of it must be the abstract thought of the engineer.

Why are we vastly better at optimization (on some problems) than our ancestors if we share the same psychology? Culture.

How is worthwhile scientific culture built up? Testing. Your car designer relies on people testing the properties of materials such as toxicity, tensile strength, melting point, chemical activity and many more. He also relies on the tests of the values of air resistance, wind tunnel simulation accuracy, gravity, likely surface friction and many many more.

The car design is built upon a mountain of tests, not many done by himself. Abstract thought can't get you very far with no tests at all.

Brian, if this definition is more useful, then why isn't that license to take over the term? More specifically, how can there be any argument on the basis of some canonical definition, while the consensus seems that we really don't know the answer yet?

It seems akin to arguing that aerodynamics isn't an appropriate basis for the definition of 'flight', just because a preconceived notion of flight includes the motion of the planets as well as that of the birds, even though the mechanisms turn out to be very different.

It sounds like you're pegging "intelligence" to mean what I'd call a "universal predictor". That is, something that can predict the future (or an unknown) given some information. And that it can do so given a variety of types of sets of unknowns, where "variety" involves more than a little hand-waving.

Therefore, something that catches a fly ball ("knowing" the rules of parabolic movement) can predict the future, but is not particularly "intelligent" if that's all it can do. It may be even a wee bit more "intelligent" if it can also predict where a mortar shell lands. It is even more "intelligent" if it predicts how to land a rocket on the moon. It is even more "intelligent" if it predicts the odds that any given cannon ball will land on a fort's walls. Etc.

I agree with Brain that this is a narrow definition of "intelligence". But that doesn't stop it from being an appropriate goal for AI at this time. That the word, "intelligence" is chosen to denote this goal seems more a result of culture than anything else. AI people go through a filter that extols "intelligence". So ... (One is reminded of many years ago when some AI thinkers had the holy grail of creating a machine that would be able to do the highest order of thinking the AI thinkers could possibly imagine: proving theorems. Coincidently, this is what these thinkers did for a living.)

Here's a thought on pinning down that word, "variety".

First, it seems to me that a "predictor" can be optimized to predict one thing very well. Call it a "tall" predictor (accuracy in Y, problem-domain-ness in X) Or it can be built to predict a lot of things rather poorly, but better than a coin. Call it a "flat" predictor. The question is: How efficient is it? How much prediction-accuracy comes out of this "predictor" given the resources it consumes? Or, using the words, "tall" and "flat" graphically, what's the surface area covered by the predictor, given a fixed amount of resources?

Would not "intelligence", as you mean it, be slightly more accurately defined as how efficient a predictor is and, uh, it's gotta be really wide or we ignore it?

"Speaking of separate names, I think you shouldn't call this 'steering the future' stuff 'intelligence.'"

I believe he generally calls it "an optimization process". His definition of intelligence was given in an earlier post, and puts more constraints on what counts.

Brian, Felix seems to understand the distinction Eliezer is making.

I regularly face the problem of wanting to refer to the process of selection acting on random mutations by some simple term. Clearly the term "evolution" won't do - since that term is already taken by a definition which fails to exclude processes such as intelligent design by genetic engineers. Probably the single best solution I have come up with is "primitive evolution".

Eliezer, I wanna tell you something that will sound offensive, please keep in mind I'm not trying to offend you...

You're making a religion out of your stuff.

Your posts are very different from Robin's - he shows specific applications of rationality, while you preach rationality as a Way. Maybe it has to do with your ethnicity: inventing religions is the #1 unique, stellar specialty of Jews. (Quick examples: Abrahamic religions, socialism, Ayn Rand's view of capitalism. Don't get offended, don't.)

Not saying your personal brand of rationality is wrong - far from it! It's very interesting, and you have taught me much. But as the blog title says, notice and overcome the bias.

Because religions have a way of becoming ugly in the long run.

Vladimir, you are slightly incorrect. Eliezer doesn't preach rationality as a Way, he preaches a Way that he claims is rationality.

And like any other preacher, he doesn't take well to people questioning and challenging him. You're likely to see your comments censored.

I agree that we don't have a precise definition of intelligence, but we do have a very rough consensus on particular examples. That consensus rejects rocks, trees, and apple pies as not intelligent. It also seems to be rejecting paperclip maximizers and happy-face tilers.

I disagree. "Superintelligent" (i.e. really powerful) paperclip maximizers are intelligent.

Your posts are very different from Robin's - he shows specific applications of rationality, while you preach rationality as a Way.

Eliezer's posts have a specific, mystical, reverent tone that I've seen in many popular science books. Was Carl Sagan hawking a religion?

Vladimir, you haven't been reading this blog for long, have you? Your comment history shows comments on only recent posts. I've done plenty of application to specifics - see e.g. the Quantum Physics Sequence. What I apply to those specifics is, of course, the Way.

Pseudo-religious style is sort of a nerd tradition as well.

If we want a robot that can navigate mazes, we could put some known pathfinding/search algorithms into it.

Or we could put a neural network in it and run it through thousands of trials with slowly increasing levels of difficulty.

In the first case it is engineered-reliable, in the second case it is learning.

On rereading the opening: "searching possible chains of causality, across learned domains, in order to find actions leading to a future ranked high in a preference ordering"

I feel like you skipped over the concept of learning. Yes we can build maze and chess searchers, because we have learned the domain.

How do we engineer something to be reliably better than us at learning? (To the point that we trust it to use it's learning to construct it's preference ordering.)

"You can't build build Deep Blue by programming a good chess move for every possible position."

Syntax error: Subtract one 'build'.

Was Carl Sagan hawking a religion?

Yes. He was trying to convince people that rationality could substitute itself for mysticism, which they highly value.

Pretty much everyone trying to "sell" an idea dip into religious memes, sooner or later -- as my now-deleted comment makes clear.

@James: If we want a robot that can navigate mazes, we could put some known pathfinding/search algorithms into it. Or we could put a neural network in it and run it through thousands of trials with slowly increasing levels of difficulty.

That evokes some loopy thinking. To wit:

It's always seemed that AI programs, striving for intelligence, can have their intelligence measured by how easy it is to get them to do something. E.g. It's easier to simply run that neural net through a bunch of trials than it is to painstakingly engineer an algorithm for a particular search problem.

So, does that mean that the definition of "intelligence" is: "How easy it is for me to get the intelligent being to do my bidding multiplied by the effect of their actions?"

Or is that a definition of "intelligence we want"? And the definition of "intelligence" is: "The ability to create "intelligence we want" and avoid "intelligence we don't want"?

And the definition of "intelligence" is: "The ability to create "intelligence we want" and avoid "intelligence we don't want"?

While a recursive definition of intelligence might please Hofstadter, I'd rather avoid it.

@Felix I started out trying to make a case that the systems we have that seem most promising are the ones that learn, in one way or another. A general intelligence has to be something that you dump training data into.

It IS easier to run the neural net through trials when you don't understand the problem and might not even know where to start. As eliezer said "Deep Blue's programmers had to know the rules of chess - since Deep Blue wasn't enough of a general AI to learn the rules by observation"

On the one hand, this is fine, we know the rules of physics close enough for the AI to design things we ask for. But it can't do real autonomous planning or self modification (and still be friendly) without a friendly preference ordering/utility function.

The way Eliezer suggests (if I understand) to create a friendly utility function/preference ordering is to have the AI analyze human brains and prefer outcomes we would prefer if we were that smart. But since we don't have the 'rules of the Brain', the AI has to learn. (unless it does a full brain simulation every time it needs to make a preference decision, but I'd rank that as a low preference.)

I'm having trouble imagining a way to engineer a learning system where we know what will come out of it without ourselves understanding the problem. True, we only need to know the end result, and not intermediate steps. But how can we know if a learning system will miss something vital on a problem that is also beyond our scope?

If you didn't know the rules of chess, you could make an AI learn the rules. But when do you bet your life that it hasn't missed something obscure but important?

Ahem: something obscure but important?

Vladimir, you haven't been reading this blog for long, have you?

Eliezer, I've lurked here for about a year. The quantum sequence was great (turned me on to many-worlds), but already pretty religious, e.g. the rationale or "it came time to break your allegiance to Science". I ate the tasty intellectual parts and mentally discarded the nasty religious parts. (For example, attacking science by attacking the Copenhagen interpretation was pretty low - most physicists don't even consider interpretations science.) Your recent posts however are all nasty, no tasty. Talmudic.

Thanks for reminding about "Is Humanism A Religion-Substitute?", it's a perfect example of what I'm talking about. You seem to be instinctively religious - you want to worship something, while e.g. for me it's just distasteful.

Religions don't go bad because they are false and stupid. Religions go bad because they live on the "ought" side of is/ought, where there is no true and false. (Cue your morality sequence.)

Vladimir,

Excellent comments and insight.

The biggest problem with being religious, in the way you have defined, is that your "oughts" start to influence your perception of what "is". We see exactly the same thing happening with traditional religions, with political religions, and here with Singularitarian religion.

The one true worthy action is to perceive the "is" without blurring our vision with the distorting filters of "oughts". This is the true "Way". But, of course, it is not nearly as popular as promising salvation, whether religious or technological.

For what it's worth, I still find the mock-mysticism stuff fairly entertaining. Hard to think of another running joke that would stay good over the same duration.

Lest we forget: http://lesswrong.com/static/imported/2008/03/27/elimonk2darker.jpg

Interesting... Reading this entry somewhat awakened my interest in computer chess... And I think I have stumled upon a nice example "inteligence as book smart" misconception.

http://en.wikipedia.org/wiki/Chess960

"Fischer believed that eliminating memorized book moves would level the playing field; as an accidental consequence, it makes computer chess programs much weaker, as they rely heavily on the opening book to beat humans."

Classic example of "bias". The reality:

"In 2005, the chess program Shredder, developed by Stefan Meyer-Kahlen from Düsseldorf, Germany, played two games against Zoltán Almási from Hungary; Shredder won 2-0."

That was only the second public match of computer vs human.

It is also worth noting that almost the equal relevance in the wiki article is given to human competitions as is to computer ones.

Now, of course, the remaining question is: "does the machine understand chinesse if it can speak better than chinaman"? :)

"If your dreams and aspirations interfere with your rationality and impartiality then you're no longer following the Way. The Way doesn't drive one off course; one rather loses sight of the Way and hence goes off course."

[The Book of Eliezer 4:24 18-20]

Quick question: How would you build something smarter, in a general sense, than yourself? I'm not doubting that it's possible, I'm just interested in knowing the specific process one would use.

Keep it brief, please. ;-)

@Silas

Quick reply: I think we will not know exactly, but that does not mean we cannot do it.

More detailed: I believe that the process will go with some sort of connectionist approach maybe mixed with some genetic algorithm voodoo. The key is reaching high complexity system that does something and starts to self-evolve (which in fact is nearly the same thing as learn).

Anyway, I believe that when all is done, we will have the same problem understanding Strong-AI mental processes as we have human brain ones.

Religions don't go bad because they are false and stupid. Religions go bad because they live on the "ought" side of is/ought, where there is no true and false. (Cue your morality sequence.)

So you're telling me I ought to stop doing that?

@Silas

I thought the heart of EY's post was here:

even if you could record and play back "good moves", the resulting program would not play chess any better than you do.

If I want to create an AI that plays better chess than I do, I have to program a search for winning moves. I can't program in specific moves because then the chess player really won't be any better than I am. [...] If you want [...] better [...], you necessarily sacrifice your ability to predict the exact answer in advance - though not necessarily your ability to predict that the answer will be "good" according to a known criterion of goodness. "We never run a computer program unless we know an important fact about the output and we don't know the output," said Marcello Herreshoff.

So the heart of the AI is something that can generate and recognize good answers. In game playing programs, it didn't take long for the earliest researchers to come up with move and position evaluators that they have been improving on ever since. There have even been some attempts at general move and position evaluators. (See work on Planner, Micro-Planner, and Conniver, which will probably lead you to other similar work.) Move generation has always been simpler in the game worlds than it would be for any general intelligence. The role of creativity hasn't been explored that much AFAICT, but it's crucial in realms where the number of options at any point are so much larger than in game worlds.

The next breakthrough will require some different representation of reality and of goals, but Eli seems to be pointing at generation and evaluation of action choices as the heart of intelligent behavior. The heart of it seems to be choosing a representation that makes generation and analysis of possible actions tractable. I'm waiting to see if EY has any new ideas on that front. I can't see how progress will be made without it, even in the face of all of EY's other contributions to understanding what the problem is and what it would mean to have a solution.

And EY has clearly said that he's more interested in behavior ("steering the future") than recognition or analysis as a characteristic of intelligence.

"We never run a computer program unless we know an important fact about the output and we don't know the output," said Marcello Herreshoff.

That's a fallacy - as I have previously explained on Overcoming Bias: we do not just run computer programs to learn things we also run them to do things.

For example, we use computer programs to automate common tasks. Say I want the numbers from 1 to 100 printed down the left side of a piece of paper. It would be reasonable to write a Python script to do that. Not because there is something unknown about the output, but because a computer and a printer can do it faster, better and neater than I can.

This seems to happen a lot in computer vision - particularly object recognition - where the prevailing paradigm involves training the system on images taken from numerous perspectives which then become exemplars for subsequent matching. The problem with this is that it just doesn't scale. Having a large number of 2D training examples, then trying to match also in 2D, works for objects which are inherently flat - like the cover of a book - but not for objects with significant 3D structure, such as a chair. For more effective recognition the system needs to capture the 3D essence of the object, not it's innumerable 2D shadows.

Tim Tyler: That's a fallacy - as I have previously explained on Overcoming Bias: we do not just run computer programs to learn things we also run them to do things. ... Not because there is something unknown about the output, but because a computer and a printer can do it faster, better and neater than I can.

That's a silly distinction to use as a basis of criticism. Would you also say that it's a fallacy to claim that we use cars to get to distant places, since we could also get there by walking?

That's a silly distinction to use as a basis of criticism. Would you also say that it's a fallacy to claim that we use cars to get to distant places, since we could also get there by walking?

The claim was that something never happens. Clearly there are counter-examples where it does indeed happen - and indeed a whole class of violations.

The car analogy is not relevant. Better to imagine that it had been claimed that humans never travel anywhere, unless they are in a car - in which case observing people travelling by bicycle would be the corresponding counter-example.

It's a stretch, but one could say that in that case you want an output with the property [is a physical system consisting of the numbers 1 to 100 prettily printed], and don't know any exact arrangement of atoms with this property.

Nick_Tarleton: Actually, Tim_Tyler's claim would still be true there, because you may want to print out that list, even if you knew some exact arrangement of atoms with that property.

However, I think Marcello's Rule is still valid there and survives Tim_Tyler's objection: in that case, what you don't know is "the sensation arising from looking at a the numbers 1 through 100 prettily printed". Even if you had seen such a list before, you probably would want to print it out unless your memory were perfect.

My claim generalizes nicely. For example, even if you ran a program for the purpose of automating a farm, and knew exactly how the farm would work, then what you don't know in that case is "the sensation of subsisting for x more days". Although Marcello's Rule starts to sound vacuous at that point.

Hey, make a squirrely objection, get a counterobjection twice as squirrely ;-)

The truly squirrely response would be to claim that a farm automation program that simply repeats an exact, preprogrammed sequence over and over is a "tape recorder" rather than a "computer program".

@Eliezer_Yudkowsky: It wouldn't be an exact sequence repeating, since the program would have to handle contingencies, like cows being uncooperative because of insufficiently stimulating conversation.

There seem to be even more objections :-(

What is unknown is not the output, but the sensation it generates;

Checking with the original wording: "We never run a computer program unless we know an important fact about the output and we don't know the output."

It seems to specify that the output is what is unknown - not the sensations that output generates in any particular observer.

An automation program that does something is not a computer program at all;

That seems fairly trivially wrong to me. Plenty of scripts that people write to do things surely are computer programs - by most people's definitions. E.g. they are written in common computer programming languages, and execute on conventional computer hardware.

To "know" the output of a program, it must be physically at hand.

That seems like a rather biblical sense of the word "know" ;-)

Do we "know" the output of a program that prints the square root of 81 before we run it? I think most would say "yes" - even if they did not have the printed output in front of them.

Why so much spirited defense of Marcello Herreshoff's inaccurate statement? ;-)

For what it's worth, Tim Tyler, I'm with you. Utility scripts count as programs in my books.

Very worthwhile points, Tim_Tyler.

First of all, the reason for my spirited defense of MH's statement is that looked like a good theory because of how concise it was, and how consistent with my knowledge of programs it was. So, I upped my prior on it and tended to see apparent failures of it as a sign I'm not applying it correctly, and that further analysis could yield a useful insight.

And I think I that belief is turning out to be true:

It seems to specify that the output is what is unknown - not the sensations that output generates in any particular observer.

But the sensations are a property of the output. In a trivial sense: it is a fact about the output, that a human will perceive it in a certain way.

And in a deeper sense, the numeral "9" means "that that someone will perceive as symbol representing the number nine in the standard number system". I'm reminded of Douglas Hofstadter's claim that definition of individual letters is an AI-complete problem because you must know a wealth of information about the cognitive system to be able to identify the full set of symbols someone will recognize as e.g. an "A".

This yields the counterintuitive result that, for certain programs, you must reference the human cognitive system (or some concept isomorphic thereto) in listing all the facts about the output. That result must hold for any program whose output will eventually establish mutual information with your brain.

Am I way off the deep end here? :-/

Further analysis might conceivably yield a useful insight - I'm only suggesting that the claim as it stands is an over-generalisation and has exceptions.

As well as automation programs, we must consider programs that are run to occupy time or burn resources, programs that are run in order to DOS-attack the computer (see The Forbin Project), pointless programs, and so on. In such cases the output may be irrelevant or non-existent, rather than unknown.

Further analysis, you say, Tim_Tyler? Could you please redirect effort away from putdowns and into finding what was wrong with the reasoning in my previous comment?

But the sensations are a property of the output. In a trivial sense: it is a fact about the output, that a human will perceive it in a certain way.

Can I ask you to taboo property.

I like to think there is some difference between a metals melting point and that it is Freds least favourite metal. I consider the second to be a property of Fred (that is how I would code it anyway), not of the metal.

Okay, fair challenge.

I agree about your metal example, but it differs significantly from my discussion of the list-output program for the non-trivial reason I gave: specifically, the output is defined by its impact on people's cognitive structure.

Look at it this way: Tim_Tyler claims that I know everything there is to know about the output of a program that spits out the integers from 1 to 100. But, when I get the output, what makes me agree that I am in fact looking at those integers? Let's say that when printing it out (my argument can be converted to one about monitor output), I see blank pages. Well, then I know something messed up: the printer ran out of ink, was disabled, etc.

Now, here's where it gets tricky: what if instead it only sorta messes up: the ink is low and so it's applied unevenly so that only parts of the numbers are missing? Well, depending on how badly it messes up, I may or may not still recognize the numbers as being the integers 1-100. It depends on whether it retains enough of the critical characteristics of those numbers for me to so recognize them.

To tie it back to my original point, what this all means is that the output is only defined with respect to a certain cognitive system: that determines whether the numbers are in fact recognizable as 9's, etc. If it's not yet clear what the difference is between this and metal's melting point, keep in mind that we can write a program to find a metal's melting point, but we can't write a program that will look at a printout and know if it retains enough of its form that a human recognizes it as any specific letter -- not yet, anyway.

the output is defined by its impact on people's cognitive structure.

Is it literally? Are you saying I couldn't send a message to someone that enabled them to print out a list of the first hundred integers without referencing a human's cognitive structure.

Sure if it went wrong they wouldn't be able to fix it. But even in that case it might be as well to send three different programs of things that do the same thing, so they could fix the incorrect one, rather than sending a full spec.

Could you please redirect effort away from putdowns and into finding what was wrong with the reasoning in my previous comment?

Not a "putdown". The whole idea looks like it needs major surgery to me - at least I can't see much of interest in it as it stands. Think you can reformulate it so it makes sense? Be my guest.

Tim_Tyler claims that I know everything there is to know about the output of a program that spits out the integers from 1 to 100. [snip printer running out of ink]

"Know" should not be read as "know with certainty". We don't know anything with certainty. Anyway, such a criticism cuts against the original claim as well - since that contained "know" as well as "don't know".

Will_Pearson: Is it literally? Are you saying I couldn't send a message to someone that enabled them to print out a list of the first hundred integers without referencing a human's cognitive structure.

Yes, that's what I'm saying. It's counterintuitive because you so effortlessly refernce others' cognitive structures. In communicating, you assume a certain amount of common understanding, which allows you to know whehter your message will be understood. In sending such a message, you rely on that information. You would have to think, "will they understand what this sentence means", "can they read this font", etc.

Tim_Tyler: The whole idea looks like it needs major surgery to me - at least I can't see much of interest in it as it stands. Think you can reformulate it so it makes sense? Be my guest.

Certainly. All you have to do is read it so you can tell me what about it doesn't make sense.

Anyway, such a criticism cuts against the original claim as well - since that contained "know" as well as "don't know".

Which contests the point how?

Yes, that's what I'm saying. It's counterintuitive because you so effortlessly refernce others' cognitive structures. In communicating, you assume a certain amount of common understanding, which allows you to know whehter your message will be understood. In sending such a message, you rely on that information. You would have to think, "will they understand what this sentence means", "can they read this font", etc.

Personally I'm thinking about communicating with machines myself, so I'm thinking something a long the lines of the following.

#include
void main()
{ int x; for( x= 1;x<101; x++){printf("%d\n",x);} }

That completely defines what I want the system to do. I do not reference other peoples thoughts.

Now it does rely on shared knowledge, the system has to have the standard libraries and a c compiler. But I could send that and still not reference humanity at all. I could send a C compiler in machine code. But that relies on me sharing knowledge with the other side on what machine code it uses.

With no shared knowledge, they cannot understand any type of definition I send. No matter if I reference human senses or not.

Genius2008's post seems a bit antagonistic. Also, it contains the idea that "AGI is only proven through construction the rest is babble and hollow claims".

We already have intelligent agents on the planet - surely an excellent and highly practical demonstration of the viability of the enterprise.