Simultaneously Right and Wrong

88 Yvain 07 March 2009 10:55PM

Related to: Belief in Belief, Convenient Overconfidence

     "You've no idea of what a poor opinion I have of myself, and how little I deserve it."

      -- W.S. Gilbert

In 1978, Steven Berglas and Edward Jones performed a study on voluntary use of performance inhibiting drugs. They asked subjects to solve certain problems. The control group received simple problems, the experimental group impossible problems. The researchers then told all subjects they'd solved the problems successfully, leaving the controls confident in their own abilities and the experimental group privately aware they'd just made a very lucky guess.

Then they offered the subjects a choice of two drugs to test. One drug supposedly enhanced performance, the other supposedly handicapped it.

There's a cut here in case you want to predict what happened.

continue reading »

Chaotic Inversion

52 Eliezer_Yudkowsky 29 November 2008 10:57AM

I was recently having a conversation with some friends on the topic of hour-by-hour productivity and willpower maintenance—something I've struggled with my whole life.

I can avoid running away from a hard problem the first time I see it (perseverance on a timescale of seconds), and I can stick to the same problem for years; but to keep working on a timescale of hours is a constant battle for me.  It goes without saying that I've already read reams and reams of advice; and the most help I got from it was realizing that a sizable fraction other creative professionals had the same problem, and couldn't beat it either, no matter how reasonable all the advice sounds.

"What do you do when you can't work?" my friends asked me.  (Conversation probably not accurate, this is a very loose gist.)

And I replied that I usually browse random websites, or watch a short video.

"Well," they said, "if you know you can't work for a while, you should watch a movie or something."

"Unfortunately," I replied, "I have to do something whose time comes in short units, like browsing the Web or watching short videos, because I might become able to work again at any time, and I can't predict when—"

And then I stopped, because I'd just had a revelation.

continue reading »

Crisis of Faith

57 Eliezer_Yudkowsky 10 October 2008 10:08PM

Followup toMake an Extraordinary Effort, The Meditation on Curiosity, Avoiding Your Belief's Real Weak Points

"It ain't a true crisis of faith unless things could just as easily go either way."
       —Thor Shenkel

Many in this world retain beliefs whose flaws a ten-year-old could point out, if that ten-year-old were hearing the beliefs for the first time.  These are not subtle errors we are talking about.  They would be child's play for an unattached mind to relinquish, if the skepticism of a ten-year-old were applied without evasion. As Premise Checker put it, "Had the idea of god not come along until the scientific age, only an exceptionally weird person would invent such an idea and pretend that it explained anything."

And yet skillful scientific specialists, even the major innovators of a field, even in this very day and age, do not apply that skepticism successfully.  Nobel laureate Robert Aumann, of Aumann's Agreement Theorem, is an Orthodox Jew:  I feel reasonably confident in venturing that Aumann must, at one point or another, have questioned his faith.  And yet he did not doubt successfullyWe change our minds less often than we think.

This should scare you down to the marrow of your bones.  It means you can be a world-class scientist and conversant with Bayesian mathematics and still fail to reject a belief whose absurdity a fresh-eyed ten-year-old could see.  It shows the invincible defensive position which a belief can create for itself, if it has long festered in your mind.

What does it take to defeat an error which has built itself a fortress?

But by the time you know it is an error, it is already defeated.  The dilemma is not "How can I reject long-held false belief X?" but "How do I know if long-held belief X is false?"  Self-honesty is at its most fragile when we're not sure which path is the righteous one.  And so the question becomes:

How can we create in ourselves a true crisis of faith, that could just as easily go either way?

continue reading »

AIs and Gatekeepers Unite!

10 Eliezer_Yudkowsky 09 October 2008 05:04PM

"Bah, everyone wants to be the gatekeeper. What we NEED are AIs."
        -- Schizoguy

Some of you have expressed the opinion that the AI-Box Experiment doesn't seem so impossible after all.  That's the spirit!  Some of you even think you know how I did it.

There are folks aplenty who want to try being the Gatekeeper.  You can even find people who sincerely believe that not even a transhuman AI could persuade them to let it out of the box, previous experiments notwithstanding.  But finding anyone to play the AI - let alone anyone who thinks they can play the AI and win - is much harder.

Me, I'm out of the AI game, unless Larry Page wants to try it for a million dollars or something.

But if there's anyone out there who thinks they've got what it takes to be the AI, leave a comment.  Likewise anyone who wants to play the Gatekeeper.

continue reading »

Shut up and do the impossible!

28 Eliezer_Yudkowsky 08 October 2008 09:24PM

Followup toMake An Extraordinary Effort, On Doing the Impossible, Beyond the Reach of God

The virtue of tsuyoku naritai, "I want to become stronger", is to always keep improving—to do better than your previous failures, not just humbly confess them.

Yet there is a level higher than tsuyoku naritai.  This is the virtue of isshokenmei, "make a desperate effort".  All-out, as if your own life were at stake.  "In important matters, a 'strong' effort usually only results in mediocre results."

And there is a level higher than isshokenmei.  This is the virtue I called "make an extraordinary effort".  To try in ways other than what you have been trained to do, even if it means doing something different from what others are doing, and leaving your comfort zone.  Even taking on the very real risk that attends going outside the System.

But what if even an extraordinary effort will not be enough, because the problem is impossible?

I have already written somewhat on this subject, in On Doing the Impossible.  My younger self used to whine about this a lot:  "You can't develop a precise theory of intelligence the way that there are precise theories of physics.  It's impossible!  You can't prove an AI correct.  It's impossible!  No human being can comprehend the nature of morality—it's impossible!  No human being can comprehend the mystery of subjective experience!  It's impossible!"

And I know exactly what message I wish I could send back in time to my younger self:

Shut up and do the impossible!

continue reading »

The Magnitude of His Own Folly

26 Eliezer_Yudkowsky 30 September 2008 11:31AM

Followup toMy Naturalistic Awakening, Above-Average AI Scientists

In the years before I met that would-be creator of Artificial General Intelligence (with a funded project) who happened to be a creationist, I would still try to argue with individual AGI wannabes.

In those days, I sort-of-succeeded in convincing one such fellow that, yes, you had to take Friendly AI into account, and no, you couldn't just find the right fitness metric for an evolutionary algorithm.  (Previously he had been very impressed with evolutionary algorithms.)

And the one said:  Oh, woe!  Oh, alas!  What a fool I've been!  Through my carelessness, I almost destroyed the world!  What a villain I once was!

Now, there's a trap I knew I better than to fall into—

—at the point where, in late 2002, I looked back to Eliezer1997's AI proposals and realized what they really would have done, insofar as they were coherent enough to talk about what they "really would have done".

When I finally saw the magnitude of my own folly, everything fell into place at once.  The dam against realization cracked; and the unspoken doubts that had been accumulating behind it, crashed through all together.  There wasn't a prolonged period, or even a single moment that I remember, of wondering how I could have been so stupid.  I already knew how.

And I also knew, all at once, in the same moment of realization, that to say, I almost destroyed the world!, would have been too prideful.

It would have been too confirming of ego, too confirming of my own importance in the scheme of things, at a time when—I understood in the same moment of realization—my ego ought to be taking a major punch to the stomach.  I had been so much less than I needed to be; I had to take that punch in the stomach, not avert it.

continue reading »

Above-Average AI Scientists

21 Eliezer_Yudkowsky 28 September 2008 11:04AM

Followup toThe Level Above Mine, Competent Elites

(Those who didn't like the last two posts should definitely skip this one.)

I recall one fellow, who seemed like a nice person, and who was quite eager to get started on Friendly AI work, to whom I had trouble explaining that he didn't have a hope.  He said to me:

"If someone with a Masters in chemistry isn't intelligent enough, then you're not going to have much luck finding someone to help you."

It's hard to distinguish the grades above your own.  And even if you're literally the best in the world, there are still electron orbitals above yours—they're just unoccupied.  Someone had to be "the best physicist in the world" during the time of Ancient Greece.  Would they have been able to visualize Newton?

At one of the first conferences organized around the tiny little subfield of Artificial General Intelligence, I met someone who was heading up a funded research project specifically declaring AGI as a goal, within a major corporation.  I believe he had people under him on his project.  He was probably paid at least three times as much as I was paid (at that time).  His academic credentials were superior to mine (what a surprise) and he had many more years of experience.  He had access to lots and lots of computing power.

And like nearly everyone in the field of AGI, he was rushing forward to write code immediately—not holding off and searching for a sufficiently precise theory to permit stable self-improvement.

In short, he was just the sort of fellow that...  Well, many people, when they hear about Friendly AI, say:  "Oh, it doesn't matter what you do, because [someone like this guy] will create AI first."  He's the sort of person about whom journalists ask me, "You say that this isn't the time to be talking about regulation, but don't we need laws to stop people like this from creating AI?"

continue reading »

Competent Elites

46 Eliezer_Yudkowsky 27 September 2008 12:07AM

Followup toThe Level Above Mine

(Anyone who didn't like yesterday's post should probably avoid this one.)

I remember what a shock it was to first meet Steve Jurvetson, of the venture capital firm Draper Fisher Jurvetson.

Steve Jurvetson talked fast and articulately, could follow long chains of reasoning, was familiar with a wide variety of technologies, and was happy to drag in analogies from outside sciences like biology—good ones, too.

I once saw Eric Drexler present an analogy between biological immune systems and the "active shield" concept in nanotechnology, arguing that just as biological systems managed to stave off invaders without the whole community collapsing, nanotechnological immune systems could do the same.

I thought this was a poor analogy, and was going to point out some flaws during the Q&A.  But Steve Jurvetson, who was in line before me, proceeded to demolish the argument even more thoroughly.  Jurvetson pointed out the evolutionary tradeoff between virulence and transmission that keeps natural viruses in check, talked about how greater interconnectedness led to larger pandemics—it was very nicely done, demolishing the surface analogy by correct reference to deeper biological details.

I was shocked, meeting Steve Jurvetson, because from everything I'd read about venture capitalists before then, VCs were supposed to be fools in business suits, who couldn't understand technology or engineers or the needs of a fragile young startup, but who'd gotten ahold of large amounts of money by dint of seeming reliable to other business suits.

One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

continue reading »

The Level Above Mine

42 Eliezer_Yudkowsky 26 September 2008 09:18AM

Followup toThe Proper Use of Humility, Tsuyoku Naritai

(At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really will be short.)

I once lent Xiaoguang "Mike" Li my copy of "Probability Theory: The Logic of Science".  Mike Li read some of it, and then came back and said:

"Wow... it's like Jaynes is a thousand-year-old vampire."

Then Mike said, "No, wait, let me explain that—" and I said, "No, I know exactly what you mean."  It's a convention in fantasy literature that the older a vampire gets, the more powerful they become.

I'd enjoyed math proofs before I encountered Jaynes.  But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments.  Maybe because Jaynes was lining up "paradoxes" that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower—power being used to overcome others.  Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection—something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion.  Of course, when you write a book, you get a chance to show only your best side.  But still.

continue reading »

That Tiny Note of Discord

16 Eliezer_Yudkowsky 23 September 2008 06:02AM

Followup toThe Sheer Folly of Callow Youth

When we last left Eliezer1997, he believed that any superintelligence would automatically do what was "right", and indeed would understand that better than we could; even though, he modestly confessed, he did not understand the ultimate nature of morality.  Or rather, after some debate had passed, Eliezer1997 had evolved an elaborate argument, which he fondly claimed to be "formal", that we could always condition upon the belief that life has meaning; and so cases where superintelligences did not feel compelled to do anything in particular, would fall out of consideration.  (The flaw being the unconsidered and unjustified equation of "universally compelling argument" with "right".)

So far, the young Eliezer is well on the way toward joining the "smart people who are stupid because they're skilled at defending beliefs they arrived at for unskilled reasons".  All his dedication to "rationality" has not saved him from this mistake, and you might be tempted to conclude that it is useless to strive for rationality.

But while many people dig holes for themselves, not everyone succeeds in clawing their way back out.

And from this I learn my lesson:  That it all began—

—with a small, small question; a single discordant note; one tiny lonely thought...

continue reading »

View more: Prev | Next