Anna, If you're talking about real dragons, the theory that made the most intuitive sense to me (I think I read it in an E.O. Wilson writing?) is that dragons are an amalgamation of things we've been naturally selected to biologically fear: snakes and birds of prey (I think rats may have also been part of the list). Dragons don't incorporate an element of them that looks like a handgun or a piping hot electric stove, probably because they're too new as threats for us to be naturally selected to fear things with those properties.

The statement the above post refers to:

http://www.singinst.org/overview/whyworktowardthesingularity

This statement seems to me to be extraordinarily (relative to the capabilities of the presumed authors) ungrounded in empiricism. All sorts of ideas in it are framed as declarative fact, when I think they should be more accurately presented as conjecture or aspirations of unknown certainty. I'm very interested in the Singularity Institute people at overcomingbias addressing these concerns directly.

Maybe the Brazilian Appeals Court was right?

http://apnews.myway.com/article/20070718/D8QEV3703.html

I'd like to lobby for a new open thread to be created weekly.

It may be a fair question of whether better outcomes result when a substantial portion of the population is taught followign directions rather than to think critically. Sort of like how the Straussians approach religion and how the armed forces approach chain of command.

Robin, good point. At the same time, there might be a large functional vs. optimal gap in the degree to which school is fulfilling its real purposes. Although the best way to optimize it might not be to brainstorm about how to get it closer to its stated purposes -so point well-taken on that end.

Great post, Eliezer (you've earned my approval). I think tied for worst school-nutured habit, along with parroting things back, is emphasis on what we think we know, as opposed to what we don't know. I think school science and history subjects would be a lot more interesting, and accurately presented, if at least equal time was given to all the problems and areas where we don't know what's going on, and for which there are various competing theories. Unfortunately one doesn't usually get this presentation of the state of things until one is working as a research assistant in college or grad school.

Nick and Eliezer, are you still Singularitarians?

http://en.wikipedia.org/wiki/Singularitarian

The idea that people are actively working to bring about self-improving, smarter-than-humanity intelligences scares me, because I think you're blind to your own ruthless selfishness (not meant pejoratively) and thus think that by creating something smarter than us (and therefore you) it can also attempt to be kind to us, as you perceive yourself to be attempting to be kind to people generally.

In contrast, I don't see either of you as Gandhi-types (here I'm referring to the archetypal elements of Gandhi's self-cultivated image, not his actual life-in-practice). It may be a hubris-derived bias that makes you think otherwise. I don't see any singulatarians performing and attempt to engage in minimal pleasurable resource use to maximize their ability to save currently existing lives. Instead I see thousands or millions of people dying daily, permanently, while leading singularians enjoy a variety of life's simple pleasures.

My prescriptive solution: more selfishness, fear, and paranoia on your end. Be thankful that you're apparently (big caveat) one of the smartest entities in apparent reality and there's apparently nothing of much greater intelligence seeking resources in your shared environment. Rather than consciously try to bring about a singularity, I think we should race against a naturally occuring singularity to understand the various existential threats to us and to minimize them.

At the same time, I think we should try to realistically assess more mundane existential threats and threats to our personal persistence, and try to minimize these too with what seems to be the best proportionate energy and effort.

But the rationalizations of why people are trying to inentionally create a self-improving intelligence smarter than humanity seem to me to be very, very weak, and could be unecessarily catastrophic to our existence.

This makes notions of representative democracy, at least in the USA, seem a bit silly:

http://andrewsullivan.theatlantic.com/the_daily_dish/2007/07/one-problem-wit.html

The link details evidence that most Americans have very low knowledge levels of the basics of American government.

Mark, alarmingly high? I don't see how that probability can be calculated as any higher than the existential threat of quantum flux or other simple, random end to our apparent reality, but I'd be interested in seeing the paper.