Once upon a time...

This is a story from when I first met Marcello, with whom I would later work for a year on AI theory; but at this point I had not yet accepted him as my apprentice.  I knew that he competed at the national level in mathematical and computing olympiads, which sufficed to attract my attention for a closer look; but I didn't know yet if he could learn to think about AI.

I had asked Marcello to say how he thought an AI might discover how to solve a Rubik's Cube.  Not in a preprogrammed way, which is trivial, but rather how the AI itself might figure out the laws of the Rubik universe and reason out how to exploit them.  How would an AI invent for itself the concept of an "operator", or "macro", which is the key to solving the Rubik's Cube?

At some point in this discussion, Marcello said:  "Well, I think the AI needs complexity to do X, and complexity to do Y—"

And I said, "Don't say 'complexity'."

Marcello said, "Why not?"

Followup to:  The Tragedy of Group Selectionism, Fake Optimization Criteria, Terminal Values and Instrumental Values, Artificial Addition, Leaky Generalizations 

"I wish to live in the locations of my choice, in a physically healthy, uninjured, and apparently normal version of my current body containing my current mental state, a body which will heal from all injuries at a rate three sigmas faster than the average given the medical technology available to me, and which will be protected from any diseases, injuries or illnesses causing disability, pain, or degraded functionality or any sense, organ, or bodily function for more than ten days consecutively or fifteen days in any year..."
            -- The Open-Source Wish Project, Wish For Immortality 1.1

There are three kinds of genies:  Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

Followup to:  Disguised Queries

In Disguised Queries, I talked about a classification task of "bleggs" and "rubes".  The typical blegg is blue, egg-shaped, furred, flexible, opaque, glows in the dark, and contains vanadium.  The typical rube is red, cube-shaped, smooth, hard, translucent, unglowing, and contains palladium.  For the sake of simplicity, let us forget the characteristics of flexibility/hardness and opaqueness/translucency.  This leaves five dimensions in thingspace:  Color, shape, texture, luminance, and interior.

Suppose I want to create an Artificial Neural Network (ANN) to predict unobserved blegg characteristics from observed blegg characteristics.  And suppose I'm fairly naive about ANNs:  I've read excited popular science books about how neural networks are distributed, emergent, and parallel just like the human brain!! but I can't derive the differential equations for gradient descent in a non-recurrent multilayer network with sigmoid units (which is actually a lot easier than it sounds).

Then I might design a neural network that looks something like this:

Followup to:  Einstein's Speed

Imagine a world much like this one, in which, thanks to gene-selection technologies, the average IQ is 140 (on our scale).  Potential Einsteins are one-in-a-thousand, not one-in-a-million; and they grow up in a school system suited, if not to them personally, then at least to bright kids.  Calculus is routinely taught in sixth grade.  Albert Einstein, himself, still lived and still made approximately the same discoveries, but his work no longer seems exceptional.  Several modern top-flight physicists have made equivalent breakthroughs, and are still around to talk.

(No, this is not the world Brennan lives in.)

One day, the stars in the night sky begin to change.

Some grow brighter.  Some grow dimmer.  Most remain the same.  Astronomical telescopes capture it all, moment by moment.  The stars that change, change their luminosity one at a time, distinctly so; the luminosity change occurs over the course of a microsecond, but a whole second separates each change.

It is clear, from the first instant anyone realizes that more than one star is changing, that the process seems to center around Earth particularly. The arrival of the light from the events, at many stars scattered around the galaxy, has been precisely timed to Earth in its orbit.  Soon, confirmation comes in from high-orbiting telescopes (they have those) that the astronomical miracles do not seem as synchronized from outside Earth.  Only Earth's telescopes see one star changing every second (1005 milliseconds, actually).

Almost the entire combined brainpower of Earth turns to analysis.

Followup to:  Artificial Addition, The Outside View's Domain

Where did I acquire, in my childhood, the deep conviction that reasoning from surface similarity couldn't be trusted?

I don't know; I really don't.  Maybe it was from S. I. Hayakawa's Language in Thought and Action, or even Van Vogt's similarly inspired Null-A novels.  From there, perhaps, I began to mistrust reasoning that revolves around using the same word to label different things, and concluding they must be similar?  Could that be the beginning of my great distrust of surface similarities?  Maybe.  Or maybe I tried to reverse stupidity of the sort found in Plato; that is where the young Eliezer got many of his principles.

And where did I get the other half of the principle, the drive to dig beneath the surface and find deep causal models?  The notion of asking, not "What other thing does it resemble?", but rather "How does it work inside?"  I don't know; I don't remember reading that anywhere.

But this principle was surely one of the deepest foundations of the 15-year-old Eliezer, long before the modern me.  "Simulation over similarity" I called the principle, in just those words.  Years before I first heard the phrase "heuristics and biases", let alone the notion of inside views and outside views.

Followup to:  Humans in Funny Suits

This fallacy gets its name from an ancient sci-fi TV show, which I never saw myself, but was reported to me by a reputable source (some guy at an SF convention).  Anyone knows the exact reference, do leave a comment.

So the good guys are battling the evil aliens.  Occasionally, the good guys have to fly through an asteroid belt.  As we all know, asteroid belts are as crowded as a New York parking lot, so their ship has to carefully dodge the asteroids.  The evil aliens, though, can fly right through the asteroid belt because they have amazing technology that dematerializes their ships, and lets them pass through the asteroids.

Eventually, the good guys capture an evil alien ship, and go exploring inside it.  The captain of the good guys finds the alien bridge, and on the bridge is a lever.  "Ah," says the captain, "this must be the lever that makes the ship dematerialize!"  So he pries up the control lever and carries it back to his ship, after which his ship can also dematerialize.

Similarly, to this day, it is still quite popular to try to program an AI with "semantic networks" that look something like this:

(apple is-a fruit)
(fruit is-a food)
(fruit is-a plant)

Followup to: Anthropomorphic Optimism, The Hidden Complexity of Wishes

Yesterday, I reprised in further detail The Tragedy of Group Selectionism, in which early biologists believed that predators would voluntarily restrain their breeding to avoid exhausting the prey population; the given excuse was "group selection".  Not only does it turn out to be nearly impossible for group selection to overcome a countervailing individual advantage; but when these nigh-impossible conditions were created in the laboratory - group selection for low-population groups - the actual result was not restraint in breeding, but, of course, cannibalism, especially of immature females.

I've made even sillier mistakes, by the way - though about AI, not evolutionary biology.  And the thing that strikes me, looking over these cases of anthropomorphism, is the extent to which you are screwed as soon as you let anthropomorphism suggest ideas to examine.

In large hypothesis spaces, the vast majority of the cognitive labor goes into noticing the true hypothesis.  By the time you have enough evidence to consider the correct theory as one of just a few plausible alternatives - to represent the correct theory in your mind - you're practically done.  Of this I have spoken several times before.

And by the same token, my experience suggests that as soon as you let anthropomorphism promote a hypothesis to your attention, so that you start wondering if that particular hypothesis might be true, you've already committed most of the mistake.

Followup to:  The Meaning of Right

Toby Ord commented:

Eliezer,  I've just reread your article and was wondering if this is a good quick summary of your position (leaving apart how you got to it):

'I should X' means that I would attempt to X were I fully informed.

Toby's a pro, so if he didn't get it, I'd better try again.  Let me try a different tack of explanation—one closer to the historical way that I arrived at my own position.

Suppose you build an AI, and—leaving aside that AI goal systems cannot be built around English statements, and all such descriptions are only dreams—you try to infuse the AI with the action-determining principle, "Do what I want."

And suppose you get the AI design close enough—it doesn't just end up tiling the universe with paperclips, cheesecake or tiny molecular copies of satisfied programmers—that its utility function actually assigns utilities as follows, to the world-states we would describe in English as:

<Programmer weakly desires 'X',   quantity 20 of X exists>:  +20
<Programmer strongly desires 'Y', quantity 20 of X exists>:  0
<Programmer weakly desires 'X',   quantity 30 of Y exists>:  0
<Programmer strongly desires 'Y', quantity 30 of Y exists>:  +60

You perceive, of course, that this destroys the world.

Followup to:  Anthropomorphic Optimism

Once upon a time there was a strange little species—that might have been biological, or might have been synthetic, and perhaps were only a dream—whose passion was sorting pebbles into correct heaps.

They couldn't tell you why some heaps were correct, and some incorrect.  But all of them agreed that the most important thing in the world was to create correct heaps, and scatter incorrect ones.

Why the Pebblesorting People cared so much, is lost to this history—maybe a Fisherian runaway sexual selection, started by sheer accident a million years ago?  Or maybe a strange work of sentient art, created by more powerful minds and abandoned?

But it mattered so drastically to them, this sorting of pebbles, that all the Pebblesorting philosophers said in unison that pebble-heap-sorting was the very meaning of their lives: and held that the only justified reason to eat was to sort pebbles, the only justified reason to mate was to sort pebbles, the only justified reason to participate in their world economy was to efficiently sort pebbles.

The Pebblesorting People all agreed on that, but they didn't always agree on which heaps were correct or incorrect.

Followup to:  Sorting Pebbles Into Correct Heaps, Invisible Frameworks

Background: There's a proposal for Friendly AI called "Coherent Extrapolated Volition" which I don't really want to divert the discussion to, right now.  Among many other things, CEV involves pointing an AI at humans and saying (in effect) "See that?  That's where you find the base content for self-renormalizing morality."

Hal Finney commented on the Pebblesorter parable:

I wonder what the Pebblesorter AI would do if successfully programmed to implement [CEV]...  Would the AI pebblesort?  Or would it figure that if the Pebblesorters got smarter, they would see that pebblesorting was pointless and arbitrary?  Would the AI therefore adopt our own parochial morality, forbidding murder, theft and sexual intercourse among too-young people?  Would that be the CEV of Pebblesorters?

I imagine we would all like to think so, but it smacks of parochialism, of objective morality.  I can't help thinking that Pebblesorter CEV would have to include some aspect of sorting pebbles.  Doesn't that suggest that CEV can malfunction pretty badly?

I'm giving this question its own post, for that it touches on similar questions I once pondered - dilemmas that forced my current metaethics as the resolution.

Yes indeed:  A CEV-type AI, taking Pebblesorters as its focus, would wipe out the Pebblesorters and sort the universe into prime-numbered heaps.

This is not the right thing to do.

That is not a bug.