I appear today on Bloggingheads.tv, in "Science Saturday: Singularity Edition", speaking with John Horgan about the Singularity.  I talked too much.  This episode needed to be around two hours longer.

One question I fumbled at 62:30 was "What's the strongest opposition you've seen to Singularity ideas?"  The basic problem is that nearly everyone who attacks the Singularity is either completely unacquainted with the existing thinking, or they attack Kurzweil, and in any case it's more a collection of disconnected broadsides (often mostly ad hominem) than a coherent criticism.  There's no equivalent in Singularity studies of Richard Jones's critique of nanotechnology - which I don't agree with, but at least Jones has read Drexler.  People who don't buy the Singularity don't put in the time and hard work to criticize it properly.

What I should have done, though, was interpreted the question more charitably as "What's the strongest opposition to strong AI or transhumanism?" in which case there's Sir Roger Penrose, Jaron Lanier, Leon Kass, and many others.  None of these are good arguments - or I would have to accept them! - but at least they are painstakingly crafted arguments, and something like organized opposition.

I'm going to try and produce summaries of the quantum physics series today or tomorrow.

Andrew Hay has produced a neat graph of (explicit) dependencies among my Overcoming Bias posts - an automatically generated map of the "Followup to" structure:

Eliezer's Post Dependencies (includes only posts with dependencies)
All of my posts (including posts without dependencies)

Subscribe here to future email notifications for when the popular book comes out (which may be a year or two later), and/or I start producing e-books:

Notifications for the rationality book, or for any other stuff I produce

(Thanks to Christian Rovner for setting up PHPList.)

Sometime in the next two weeks, I need to get at least one Powerpoint presentation of mine re-produced to professional standards of graphic design.  Ideally, in a form that will let me make small modifications myself.  This is likely to lead into other graphic design work on producing the ebooks, redesigning my personal website, creating Bayesian Conspiracy T-shirts, etc.

I am not looking for an unpaid volunteer.  I am looking for a professional graphic designer who can do sporadic small units of work quickly.

I'm in Oxford right now, for the Global Catastrophic Risks conference.

There's a psychological impact in walking down a street where where any given building might be older than your whole country.

Toby Ord and Anders Sandberg pointed out to me an old church tower in Oxford, that is a thousand years old.

At the risk conference I heard a talk from someone talking about what the universe will look like in 10¹⁰⁰ years (barring intelligent modification thereof, which he didn't consider).

The psychological impact of seeing that old church tower was greater.  I'm not defending this reaction, only admitting it.

I haven't traveled as much as I would travel if I were free to follow my whims; I've never seen the Pyramids.  I don't think I've ever touched anything that has endured in the world for longer than that church tower.

A thousand years...  I've lived less than half of 70, and sometimes it seems like a long time to me.  What would it be like, to be as old as that tower?  To have lasted through that much of the world, that much history and that much change?

I've been feeling burned on Overcoming Bias lately, meaning that I take too long to write my posts, which decreases the amount of recovery time, making me feel more burned, etc.

So I'm taking at most a one-week break.  I'll post small units of rationality quotes each day, so as to not quite abandon you.  I may even post some actual writing, if I feel spontaneous, but definitely not for the next two days; I have to enforce this break upon myself.

When I get back, my schedule calls for me to finish up the Anthropomorphism sequence, and then talk about Marcus Hutter's AIXI, which I think is the last brain-malfunction-causing subject I need to discuss.  My posts should then hopefully go back to being shorter and easier.

Hey, at least I got through over a solid year of posts without taking a vacation.

FYI all:  The Singularity Summit 2008 is coming up, 9am-5pm October 25th, 2008 in San Jose, CA.  This is run by my host organization, the Singularity Institute.  Speakers this year include Vernor Vinge, Marvin Minsky, the CTO of Intel, and the chair of the X Prize Foundation.

Before anyone posts any angry comments: yes, the registration costs actual money this year.  The Singularity Institute has run free events before, and will run free events in the future.  But while past Singularity Summits have been media successes, they haven't been fundraising successes up to this point.  So Tyler Emerson et. al. are trying it a little differently.  TANSTAAFL.

Lots of speakers talking for short periods this year.  I'm intrigued by that format.  We'll see how it goes.

Followup to:  Affective Death Spirals, My Wild and Reckless Youth

My parents always used to downplay the value of intelligence.  And play up the value of—effort, as recommended by the latest research?  No, not effort.  Experience.  A nicely unattainable hammer with which to smack down a bright young child, to be sure.  That was what my parents told me when I questioned the Jewish religion, for example.  I tried laying out an argument, and I was told something along the lines of:  "Logic has limits, you'll understand when you're older that experience is the important thing, and then you'll see the truth of Judaism."  I didn't try again.  I made one attempt to question Judaism in school, got slapped down, didn't try again.  I've never been a slow learner.

Whenever my parents were doing something ill-advised, it was always, "We know better because we have more experience.  You'll understand when you're older: maturity and wisdom is more important than intelligence."

If this was an attempt to focus the young Eliezer on intelligence uber alles, it was the most wildly successful example of reverse psychology I've ever heard of.

But my parents aren't that cunning, and the results weren't exactly positive.

Followup to:  My Childhood Death Spiral

Yesterday I covered the young Eliezer's affective death spiral around something that he called "intelligence".  Eliezer₁₉₉₆, or even Eliezer₁₉₉₉ for that matter, would have refused to try and put a mathematical definition—consciously, deliberately refused.  Indeed, he would have been loath to put any definition on "intelligence" at all.

Why?  Because there's a standard bait-and-switch problem in AI, wherein you define "intelligence" to mean something like "logical reasoning" or "the ability to withdraw conclusions when they are no longer appropriate", and then you build a cheap theorem-prover or an ad-hoc nonmonotonic reasoner, and then say, "Lo, I have implemented intelligence!"  People came up with poor definitions of intelligence—focusing on correlates rather than cores—and then they chased the surface definition they had written down, forgetting about, you know, actual intelligence.  It's not like Eliezer₁₉₉₆ was out to build a career in Artificial Intelligence.  He just wanted a mind that would actually be able to build nanotechnology.  So he wasn't tempted to redefine intelligence for the sake of puffing up a paper.

Looking back, it seems to me that quite a lot of my mistakes can be defined in terms of being pushed too far in the other direction by seeing someone else stupidity:  Having seen attempts to define "intelligence" abused so often, I refused to define it at all.  What if I said that intelligence was X, and it wasn't really X?  I knew in an intuitive sense what I was looking for—something powerful enough to take stars apart for raw material—and I didn't want to fall into the trap of being distracted from that by definitions.

Similarly, having seen so many AI projects brought down by physics envy—trying to stick with simple and elegant math, and being constrained to toy systems as a result—I generalized that any math simple enough to be formalized in a neat equation was probably not going to work for, you know, real intelligence.  "Except for Bayes's Theorem," Eliezer₂₀₀₀ added; which, depending on your viewpoint, either mitigates the totality of his offense, or shows that he should have suspected the entire generalization instead of trying to add a single exception.

Followup to:  My Best and Worst Mistake

My father used to say that if the present system had been in place a hundred years ago, automobiles would have been outlawed to protect the saddle industry.

One of my major childhood influences was reading Jerry Pournelle's A Step Farther Out, at the age of nine.  It was Pournelle's reply to Paul Ehrlich and the Club of Rome, who were saying, in the 1960s and 1970s, that the Earth was running out of resources and massive famines were only years away.  It was a reply to Jeremy Rifkin's so-called fourth law of thermodynamics; it was a reply to all the people scared of nuclear power and trying to regulate it into oblivion.

I grew up in a world where the lines of demarcation between the Good Guys and the Bad Guys were pretty clear; not an apocalyptic final battle, but a battle that had to be fought over and over again, a battle where you could see the historical echoes going back to the Industrial Revolution, and where you could assemble the historical evidence about the actual outcomes.

On one side were the scientists and engineers who'd driven all the standard-of-living increases since the Dark Ages, whose work supported luxuries like democracy, an educated populace, a middle class, the outlawing of slavery.

On the other side, those who had once opposed smallpox vaccinations, anesthetics during childbirth, steam engines, and heliocentrism:  The theologians calling for a return to a perfect age that never existed, the elderly white male politicians set in their ways, the special interest groups who stood to lose, and the many to whom science was a closed book, fearing what they couldn't understand.

And trying to play the middle, the pretenders to Deep Wisdom, uttering cached thoughts about how technology benefits humanity but only when it was properly regulated—claiming in defiance of brute historical fact that science of itself was neither good nor evil—setting up solemn-looking bureaucratic committees to make an ostentatious display of their caution—and waiting for their applause.  As if the truth were always a compromise.  And as if anyone could really see that far ahead.  Would humanity have done better if there'd been a sincere, concerned, public debate on the adoption of fire, and commitees set up to oversee its use?

Followup to:  My Childhood Death Spiral, Raised in Technophilia

My Childhood Death Spiral described the core momentum carrying me into my mistake, an affective death spiral around something that Eliezer₁₉₉₆ called "intelligence".  I was also a technophile, pre-allergized against fearing the future.  And I'd read a lot of science fiction built around personhood ethics—in which fear of the Alien puts humanity-at-large in the position of the bad guys, mistreating aliens or sentient AIs because they "aren't human".

That's part of the ethos you acquire from science fiction—to define your in-group, your tribe, appropriately broadly.  Hence my email address, sentience@pobox.com.

So Eliezer₁₉₉₆ is out to build superintelligence, for the good of humanity and all sentient life.

At first, I think, the question of whether a superintelligence will/could be good/evil didn't really occur to me as a separate topic of discussion.  Just the standard intuition of, "Surely no supermind would be stupid enough to turn the galaxy into paperclips; surely, being so intelligent, it will also know what's right far better than a human being could."

Until I introduced myself and my quest to a transhumanist mailing list, and got back responses along the general lines of (from memory):

Followup to:  My Childhood Death Spiral, My Best and Worst Mistake, A Prodigy of Refutation

"There speaks the sheer folly of callow youth; the rashness of an ignorance so abysmal as to be possible only to one of your ephemeral race..."
        —Gharlane of Eddore

Once upon a time, years ago, I propounded a mysterious answer to a mysterious question—as I've hinted on several occasions.  The mysterious question to which I propounded a mysterious answer was not, however, consciousness—or rather, not only consciousness.  No, the more embarrassing error was that I took a mysterious view of morality.

I held off on discussing that until now, after the series on metaethics, because I wanted it to be clear that Eliezer₁₉₉₇ had gotten it wrong.

When we last left off, Eliezer₁₉₉₇, not satisfied with arguing in an intuitive sense that superintelligence would be moral, was setting out to argue inescapably that creating superintelligence was the right thing to do.

Well (said Eliezer₁₉₉₇) let's begin by asking the question:  Does life have, in fact, any meaning?