The URL to the anime fanfiction seems to be worse than broken. My browser doesn't even say what you wrote, just that it's "illegal".

(page source: )

Aw great, now my post is broken.

Good post, more or less fully agree, except I still don't get this bit: "In a safer world, I would have wished for my parents to have hidden that book better..."

Why? Not the "in a safer world" part, I get why you put that clause there, I think. It's the rest of it. Why would you have rather not learned the things in that book at an early age? If you don't mind explaining that one, I'm genuinely curious why you would have preferred, (in a safer world) not having learned that at an early age.

(Also, "I don't know if my parents ever thought about the child-adult dichotomy when they weren't talking to me.", well, presumably easy enough, to at least to a first approximation, to find out. (If you're interested) Ask them, or even point them right at this post and see/ask for their reaction.)

So if I want to help with issues I believe are important and neglected such as people dying of old age, and if I think that the way you are advocating, superintelligence, makes sense and is the best possible way I've seen so far, I shouldn't try to signal rationality by prattling here, in order to attract attention, but rather, what ? (Especially if I'm not good at math)(the temptation remains great anyway). At any rate, an excellent post, I matured a bit by reading it.

"at age nine or something silly like that, showing my father a diagram full of filled circles and trying to convince him that the indeterminacy of particle collisions was because they had a fourth-dimensional cross-section and they were bumping or failing to bump in the fourth dimension."

You were a lucky child. You got real science at age nine! I had to wait until I was 15 or 16...

Good heavens, Eliezer. Rationality is Serious Business. Grow up and stop acting like you... enjoy it or something!

Psy-Kosh: My guess is that most learning is deeper and more authentic if it is from one's own experience. Eliezer seems to particularly prize personal learning, favoring secrecy in science in his idea of paradise.

"To be concerned about being grown up, to admire the grown up because it is grown up, to blush at the suspicion of being childish; these things are the marks of childhood and adolescence. And in childhood and adolescence they are, in moderation, healthy symptoms. Young things ought to want to grow. But to carry on into middle life or even into early manhood this concern about being adult is a mark of really arrested development. When I was ten, I read fairy tales in secret and would have been ashamed if I had been found doing so. Now that I am fifty I read them openly. When I became a man I put away childish things, including the fear of childishness and the desire to be very grown up." - C.S. Lewis

Tiiba, fixed.

Roko, this was not real science.

Info, write Michael Vassar.

Psy, if the average childhood were less dangerous and happier, I'd rather not be told how it goes in such detail as to take out the spontaneity, which really did happen in my case.

Eliezer: ah, okay. Didn't realize you intended the "and happier" bit, though you meant that even if childhood development with all the nasty bits were still in place. ie, thought you just meant something along the lines of "there's stuff that really needs to get done, pronto, in this world, so it's a good thing I did things that sped up my climb to rationality and so on so I could get to work on them."

Eliezer, are you familiar with Carol Dweck's research on intelligence, or has that corner of psychology eluded you? It matches up very closely with what you say here about maturity. Dweck says: some people (like your parents on maturity) have an "entity theory" of intelligence - they think of it as something fixed that you either have or you don't - while others (like you on maturity) have an "incremental theory" - they think of it as continually developing. Incremental theorists tend to learn better and be more eager to face challenges, while entity theorists are more threatened by challenges and care more about signaling that they have intelligence. More here.

Entity views may be a common source of bias, with intelligence and other qualities that people value.

I'll bet you avoided your vision of a "teenager" but so did a million other people. All of these implications of teenage behavior or "inherent" concerns of the teen age are such a bust because so many people recognize it that the commonality of being concerned with it becomes a bore. You could have avoided being "that" teenager but so did everyone else.

This reminds me of I believe it was Robin's post which cautioned us about interesting writing styles and anecdotes since it seems to subtract from how accurately people recall the main points of a piece of writing. From reading your blog, and others, I have been inspired to take the interesting and informal writing approach when talking about high-level concepts.

I think that perhaps in any one piece it may be a negative factor, especially if the anecdotes and cliches aren't completely pertinent to the topic. Nevertheless, I think on a larger scale that it makes sense to keep things as accessible and entertaining as possible, since that will draw more people in. Freakonomics, for all its numerous flaws, probably drew more interest in the field of economics than the illustrious General Theory of Employment, Interest, and Money.

I really liked the end where you say, "And so I take a certain dark delight in quoting anime fanfiction at people who expect me to behave like a wise sage of rationality. Why should I pretend to be mature when practically every star in the night sky is older than I am?"

Yet, you still think you should build an Artificial General Intelligence. That seems very much like childish hubris, and yet you seem to simultaneously show humility. Or is it just that you have a really low opinion of everyone else's intelligence and rationality? I wouldn't blame you. Neither would they, since statistics show the majority of people believe that they are above average intelligence.

I think all humans are walking balls of contradictions, so it isn't really surprising that you display both hubris and humility. I can also honestly say I know much much less about AI, physics, neuropsychology, and bayesian probability theory than you, so I can't say one way or another whether AGI is possible or desirable, nor could I if I spent years researching it. I tend to avoid thinking about it generally for that reason.

Just something to think about.... but you probably already have. Smarty pants.

What a wonderful, thoughtful post. As always.

I am reminded of The Moody Blues' lyrics, "With the eyes of a child"

With the eyes of a child You must come out and see That your world's spinning 'round And through life you will be A small part Of a hope Of a love That exists In the eyes of a child you will see

Edward, how is it arrogant to want to contribute to science?

I was being forced to memorize and recite

Without getting into the theological aspects, a good technique as part of learning a second language is memorizing a text, particularly one with a poetic structure (like many prayers), even before it can be fully understood.

We were given a transliteration, but not a translation. I asked what the prayer meant. I was told that I didn't need to know

That is an extremely unusual experience, except as a temporary stage of learning. In most Jewish circles, reading Hebrew in the original script is considered important, and it is believed that one does need to understand what texts mean.

Infotropism: Michael (dot) Vassar at gmail

Eliezer: You really should tell people how to reach me as well as telling them that they should do so, either with my email or a link to SIAI's "about us" page, which now has my contact info.

Tiiba,

There is a difference between science and technology. Science is just empirical knowledge and the process of acquiring it. Technology is applied science. It shows hubris to actually think you can build something smarter than yourself in a way that has a reasonable chance of avoiding terrible unforeseen consequences.

That doesn't mean he is wrong when he argues AGI is just on the horizon and it is better that someone who cares about friendliness implements it first. I don't know enough to say one way or another.

Hubris is probably one of the traits that has propelled humanity to it's current stage of civilization. It is debatable whether that was a good thing. Certainly not from a negative utilitarian perspective, but definitely from an aggregate utilitarian calculus.

Now that we are at this stage of human civilization, all we can do is continue onward. Humans taking their destiny into their own hands has the potential to solve the problems of previous attempts to do just that.

It isn't really debatable that the stories of Faust and the Tower of Babel, etc have a point, it is just a matter of figuring out to what degree. The whole idea of FAI inherently gives this idea some credence, but it says that since someone is creating the tech it might as well be us since we care about Friendliness. Oh yea, and by the way you get Singularities and stuff if we don't all die.

...never tried a single drug

I'm going to presume you've drank tea, or taken medicine, and under that presumption I can say 'Yes you did'. It's just that the drugs you chose were the ones that adults in your culture had decided were safe... things like caffeine say. Had you grown up in a mormon culture or Amish culture, you might not be able to write the same thing you just did, so isn't what you just wrote an accident of birth rather than a conscious choice about your use of particular chemical structures inside your body?

I would imagine that by choice of locale, you may have passively taken nicotine, too, albeit in small quantities.

, never lost control to hormones...

.... really? Never got angry then, or too depressed to work? Crikey. Or do you mean you only lost control in the way that your parents and culture approved of; again, nothing more than an accident of birth?

I've rarely heard "You'll understand when you're older" on questions of simple fact. Usually, it's uttered when someone who claims to be altruistic points out someone else's actions are harmful. The Old Cynic then tells the Young Idealist "I used to be like you, but then I realized you've got to be realistic, you'll understand when you're older that you should be more selfish.". But they never actually offer an object-level argument, or even seem to have changed their minds for rational reasons - it looks like the Selfishness Fairy just changed their terminal values as they grew older. That may be the case; it may also be sour grapes bias: when they realized their altruism could never have as big an effect as it ought to, they decided altruism wasn't right after all. The best defense I can come up with is: If your moral intuitions change, especially change in a way you've previously noticed as "maturing", only trust them if your justifications for it would convice your past self as their most idealistic.

Is this "stupid teeneager" thing real, or just a stereotype that sells books? I've seen teenagers drink and drive; they don't look like they do it to look adult. I've tried some drugs and turned others down, and the only things that (I'm aware) factored were what I could learn from the experience, how pleasant it would be, and the risks. I consciously ignored peer pressure - as for looking mature, I simply didn't even consider it could be a criterion any more than the parity of my number of nose hair.

"When I became a man I put away childish things, including the fear of childishness and the desire to be very grown up." - C.S. Lewis

This is a near neighbor concept that I consider important to distinguish. I don't think this is a case where it's a good idea to embrace the contradiction of calling it childish to worry about childishness. Then you get into slap-fights about whose worry over maturity signals greater immaturity.

You should just be asking some other question instead - cutting through on some other question that isn't about maturity or childishness.

I quoted "Shinji and Warhammer40k" first and foremost because it had a couple of good quotes, worth adding to my quotes file and then worth selecting. The dark delight in quoting anime fanfiction comes after the simpler deed of asking "Is this a good quote or not?"

I too, never wanted to be a stupid teenager, although I never read any books about it. I never smoked, never drank, never used illegal drugs. But I also thought my parents didn't love me. Interesting partial overlap there.

Also, unless a new star has appeared in the last 38 years less than 38 lightyears away, there's no star in the night sky younger than you are, or me, or probably anyone else on Earth.

Like you, I feel maturity is a journey, one that too many people seem to give up very early. I notice at dating sites that some women write "I don't want a man who refers to himself as a 'work in progress'". I understand what they mean - they don't want a man who doesn't know what he wants in life, who doesn't have career goals or a purpose. But what it says (to me) is 'someone who is done changing, learning and growing'.

Which is terrible.

This is a near neighbor concept that I consider important to distinguish. I don't think this is a case where it's a good idea to embrace the contradiction of calling it childish to worry about childishness. Then you get into slap-fights about whose worry over maturity signals greater immaturity.

Still, that worry over childishness is self-undermining seems like a decent point.

Actually... one reason I was careful to draw the distinction, is that the above (as it so happens) strikes me as a childish point.

Like, it was the sort of thing I never said to my parents as a teenager because it would have been too much like what a "teenager" (albeit a clever one who'd read Godel, Escher, Bach) would have said to his parents.

Is worry over appearing childish self-undermining? In general, no; it is self-undermining only if we assume that children and only children are concerned with appearing adult. As it happens, most adults are concerned with appearing adult. Are they wrong to do so? That depends on why, I'd say. A concern with appearing adult as a strictly pragmatic matter of public relations, coupled with a total unconcern over actually being adult, could well be healthy and rational in certain cases.

Of course, having actually been maneuvered into a position where you have to argue that you aren't childish, the argument itself will seem childish; but this is a matter of being placed in a clever trap, not a matter of the audience making valid Bayesian inferences.

...okay, I've probably been reading too much about logic today.

Regarding Judiasm: my father had a similar experience as a child. He assumed that the Hebrew passages he was given must have been great and profound because, well, they came from God. When he finally did read a translation, he was annoyed because they were the kind of stupid banalities that any idiot could have thought up. (My father is an atheist.)

This is sort of a tangential observation, but bear with me. This business of teenagers acting 'mature' by doing stupid, immature things is fostered by 'middle school'. By 'middle school', I mean the quite bizarre institution we have of shoving all of our pubescent offspring into an age cohort, marching them around through artificial structured activities in which they have only the company of other pubescents and a few exasperated chaperones (aka teachers, administrators, etc.) while cutting them off from either the care of younger children or access to the normal adult world.

If preschool is where our kids go to pick up viral diseases, middle school is where we send our kids to be infected with dumb social models to conform to.

End rant ;) Thanks for the thoughtful post.

You're right, dang it, I fell for the clever trap.

You strike me as more arrogant than Robin. I'm not sure why. Certainly arrogance does not help in overcoming bias.

If you quote Shakespeare, we can all see the name and might think, well, that is probably wise and well expressed. If you quote anime fanfic, we might see the source and decide that was probably silly and probably badly expressed;

or consider it a quote with Eliezer Yudkowsky's imprimatur, which might be arrogant of Eliezer, but actually I think worthwhile, that Eliezer values an idea is for me something in favour of that idea;

or I could just judge the quote myself, see what I think of it, see what good I could strain out of it.

Eliezer is my teacher, but I do permit myself to disagree with him.

" Certainly arrogance does not help in overcoming bias."

On the contrary; when dealing with deep-seated, common, and possibly hard-wired cognitive biases, I'd say it actually requires a certain degree of hubris to even consider attempting to overcome such.

that's quite an astounding amount of adults partaking in supposedly childish things if everyone you talk to recognizes youre quoting anime

So, Eliezer, Because some ill equipped teacher refused to answer a good question you broke with a time honored tradition that is singularly responsible for the intellectual achievements of the Jewish people? Somehow, we're supposed to see this as rational?

This is quite a brilliant and extremely interesting analysis. Thank you!

I'm not quite sure what you mean by "mere" signaling. If visible feature F did not correlate with hard to observe character C, then F could not signal C. Of course the correlation isn't perfect, but why doesn't it make sense to choose F if you want people to believe you have C? Are you saying you didn't really care what people thought of your maturity?

Robin, I think I've made a pretty good case (by word and deed) that I don't care much about variance in my perceived maturity so long as it stays within certain upper and lower bounds, below Infallible Guru and above 12-Year-Old On 4chan. This reflects a certain increase in maturity on my part; I used to care less.

I'd think that signals can easily come completely unbound from what they're supposed to signal. All this requires is that public image either lag the reality, or that public image be not completely based on reality, or that people are mistaken about what most other people think. Is "signal" by convention used only to indicate cases where someone is manipulating an indicator that retains actual current correlation even after the manipulability is taken into account? If so I'll need another term to indicate what people do when they display a characteristic X that they believe others will associate to Y. I understand that there are cases where you can solve for signaling equilibria assuming full information, but I did not realize this was part of the definition of "signaling" as such.

Eliezer, in most signaling theories that economists construct the observers of signals are roughly making reasonable inferences from the signals they observe. If someone proposed to us that people take feature F as signaling C, but in fact there is no relation between F and C, we would want some explanation for this incredible mistake, or at least strong evidence that such a mistake was being consistently made.