I don't understand what you mean by "intelligent," "living systems," "alive," or "intrinsic goals" (so I don't understand what any of these statements means in enough detail to determine what it would mean to falsify them). How much of the Sequences have you read? In particular, have you read through Mysterious Answers to Mysterious Questions or 37 Ways That Words Can Be Wrong?

Do goals always have to be consciously chosen? When you have simple if-then clauses, such as "if (stimulusOnLips) then StartSuckling()", doesn't that count as goal-fulfilling behavior? Even a sleeping human is pursuing an endless stream of maintenance tasks, in non-conscious pursuance of a goal such as "maintain the body in working order". Does that count?

I can see "goal" being sensibly defined either way, so it may be best not to insist on "must be consciously formulated" for the purposes of this post, then move on.

To be intelligent, a system has to have goals - it has to be an agent. (I don't think this is controversial).

Is a newborn baby human, or a human of any age who is asleep, intelligent by this definition?

Try reading the sequences all the way through. You'll find that you make a lot of common assumptions and mistakes that make the argument weaker than you'd like.

This post inspired me to work on my Mandarin study habits - I've been stuck in a low intermediate plateau for a while, and not sure how to advance. I just started to work on this mindmap, http://www.mindmeister.com/maps/show/98440507, based on the ideas in this article.

I've also recently started following GTD (the productivity system), which emphasises choosing specific actions to follow, rather than big and vague projects. I think this article's approach is similar.

A prima facie case against the likelihood of a major-impact intelligence-explosion singularity:

Firstly, the majoritarian argument. If the coming singularity is such a monumental, civilization-filtering event, why is there virtually no mention of it in the mainstream? If it is so imminent, so important, and furthermore so sensitive to initial conditions that a small group of computer programmers can bring it about, why are there not massive governmental efforts to create seed AI? If nothing else, you might think that someone could exaggerate the threat of the singularity and use it to scare people into giving them government funds. But we don’t even see that happening.

Second, a theoretical issue with self-improving AI: can a mind understand itself? If you watch a simple linear Rube Goldberg machine in action, then you can more or less understand the connection between the low- and the high-level behavior. You see all the components, and your mind contains a representation of those components and of how they interact. You see your hand, and understand how it is made of fingers. But anything more complex than an adder circuit quickly becomes impossible to understand in the same way. Sure, you might in principle be able to isolate a small component and figure out how it works, but your mind simply doesn’t have the capacity to understand the whole thing. Moreover, in order to improve the machine, you need to store a lot of information outside your own mind (in blueprints, simulations, etc.) and rely on others who understand how the other parts work.

You can probably see where this is going. The information content of a mind cannot exceed the amount of information necessary to specify a representation of that same mind. Therefore, while the AI can understand in principle that it is made up of transistors etc., its self-representation necessary has some blank areas. I posit that the AI cannot purposefully improve itself because this would require it to understand in a deep, level-spanning way how it itself works. Of course, it could just add complexity and hope that it works, but that’s just evolution, not intelligence explosion.

So: do you know any counterarguments or articles that address either of these points?