So is the reason I should believe this space of minds-in-general exists at all going to come in a later post?

I can certainly agree that you rely on this sort of reasoning a lot. But I don't think what you do is much of an improvement over what you're criticizing. You just take words and make "surface analogies" with "cognitive algorithms." The useful thing about these "cognitive algorithms" is that, being descriptions of "deep causes" (whatever those are) rather than anything we know to actually exist in the world (like, say, neurons), you can make them do whatever you please with total disregard for reality.

Saying that a neural network never gets at "intelligence" is little different from saying the descriptions of biology in textbooks never capture "life." Without a theory of "life" how will we ever know our biological descriptions are correct? The answer is as blatantly obvious as it is for neural networks by comparing them to actual biological systems. We call this "science." You may have heard of it. Of course, you could say, "What if we didn't have biology to compare it too, how then would you know you have the correct description of life?" But... well, what to say about that? If there were no biology nobody would talk about life. Likewise, if there were no brains, nobody would be talking about intelligence.

You essentially posit a "decision algorithm" to which you ascribe the sensations most people attribute to free will. I don't think this is helpful and it seems like a cop-out to me. What if the way the brain makes decisions doesn't translate well onto the philosophical apparatus of possibility and choice? You're just trading "suggestively named LISP tokens" for suggestively named algorithms. But even if the brain does do something we could gloss in technical language as "making choices among possibilities" there still aren't really possibilities and hence choices.

What it all comes down to, as you acknowledge (somewhat), is redefining terms. But if you're going to do that, why not say, "none of this really matters, use language how you will"? Actually, a lot of your essays have these little disclaimers at the end, where you essentially say "at least that's how I choose to use these words." Why not headline with that?

There are basically three issues with any of these loaded terms - free will, choice, morality, consciousness, etc - that need to be addressed: (1) the word as a token and whether we want to define it and how; (2) matters the "common folk" want reassurance on, such as whether they should assume a fatalistic outlook in the face of determinism, whether their neighbors will go on killing sprees if morality isn't made out of quarks, etc; (3) the philosophical problem of free will, problem of morality, etc.

Philosophers have made a living trying to convince us that their abstract arguments have some relevance to the concerns of the common man and that if we ignore them we're being insensitive or reductionist and are guilty of scientism and fail to appreciate the relevance of the humanities. That's egregious nonsense. Really these are three entirely separate issues. I get the impression that you actually think these problems are pseudo-problems but at the same time you tend to run issues 2 and 3 together in your discussions. Once you separate them out, though, I think the issues become trivial. It's obvious determinism shouldn't make us fatalistic because we weren't fatalistic before and nothing has changed, it's obvious we won't engage in immoral behavior if morals aren't "in the world" since we weren't immoral before and nothing has changed, etc.

michael vassar,

I think you misunderstand me. I'm not being cynical; I'm trying to demonstrate that moral dilemmas and moral deliberation aren't empirically established. I tried to do this, first, by pointing out that what most people consider the subject of morality differs substantially from the subject of academic philosophers and, second, by arguing that the type of moral reasoning found in philosophy isn't found in society at large and doesn't influence it. People really do heroically rescue orphans from burning buildings in real life and they do it without viewing the situation as a moral dilemma and without moral deliberation. I don't think a world where moral philosophy turns out to be perfectly worthless is necessarily a bad one.

The type of possibility you describe is just a product of our ignorance about our own or others psychology. If I don't understand celestial mechanics I might claim that Mars could be anywhere in its orbit at any time. If somebody then came along and taught me celestial mechanics I could then argue that Mars could still be anywhere if it wanted to. This is just saying that Mars could be anywhere if Mars was different. It gets you exactly nothing.

michael vassar,

I'm skeptical as to whether the affirmed moralities play a causal role in their behavior. I don't think this is obvious. Cultures that differ in what we call moral behavior also differ in culinary tastes but we don't think one causes the other; it's possible that they have their behaviors and they have their explanations of their behaviors and the two do not coincide (just as astrology doesn't coincide with astronomy). I'm also therefore skeptical that changes over time are caused by moral deliberation; obviously if morality plays no causal role in behavior it cannot change behavior.

What anthropologists call moral behavior and what most non-philosophers would recognize as moral behavior tends to coincide with superstitions more than weighty philosophical issues. Most cultures are very concerned with what you eat, how you dress, who you talk to, and so forth, and take these to be moral issues. Whether one should rescue a drowning child if one is a cancer researcher is not as big a concern as who you have sex with and how you do it. How much genuine moral deliberation is really going on in society? How much influence do those who engage in genuine moral deliberation (i.e., moral philosophers) have on society? I think the answers are close to "none" and "not at all."

I agree that determinism doesn't undermine morality in the way you describe. I remain, however, a moral skeptic (or, perhaps more accurately, a moral eliminativist). I'm skeptical that moral dilemmas exist outside of thought experiments and the pages of philosophy books and I'm skeptical that moral deliberation achieves anything. Since people are bound to play out their own psychology, and since we're inherently social animals and exist in a social environment, I find it unlikely that people would behave substantially different if we eliminated "morality" from our concept space. In that respect I think morality is an epiphenomenon.

Some people want to take part of our psychology and label it "morality" or take the sorts of diplomacy that lead us to cooperate for our mutual benefit and label it "morality" but they're essentially moral skeptics. They're just flexible with labels.

I picked up an intuitive sense that real thinking was that which could force you into an answer whether you liked it or not, and fake thinking was that which could argue for anything.

This is very dangerous. I think a great example of its danger is Colin McGinn (popularizer of mysterianism) in his The Making of a Philosopher. He says that what attracted him to philosophy was the ability to reason ones way to contrarian opinions. Being forced to an answer itself has an appeal. This is a major problem in the transhumanist and libertarian communities, for example, where bullet biting is much more highly regarded than having your facts straight.

mtraven,

Got a better one?

Biology and physics. Google Tim Van Gelder for a philosophical perspective on the benefits of using dynamics to explain cognition. I think he has papers online.

Presumably your brain is processing symbols right now, as your read this.

I think there's an important distinction between being able to manipulate symbols and engaging in symbol processing. After all, I can use a hammer, but nobody thinks there's hammers in my brain.

Caledonian,

But computer programmers don't need to understand the hardware, either. Do you think they crack open metallurgy, electronics, and applied physics textbooks to accomplish their goals?

Computers are specifically designed so that we don't have to understand the hardware. That's why I said it's spurious to call anything but an artifact a computer. You don't need to understand the underlying physics because engineers have carefully designed the system that way. You don't have to understand how your washing machine or your VCR works either.

If you don't need to understand every level of hardware to manipulate electronic computational devices, why do you think anyone would need to understand the physics all the way down to deal with the mind?

I don't think we need to understand the physics all the way down in a practical sense. We've already built our way up from physics through chemistry to molecular biology and the behavior of the cell. We can talk about the behavior of networks of cells too. The difference is that it's the underlying physical properties that make this abstraction possible whereas, in a computer, the system has been specifically designed to have implementation layers with reference to a set of conventions. In a loose sense, it's accurate to say we understand the physics all the way down in a biological system, because the fact of abstraction is a part of the system (i.e., the molecules interact in a way that allows us to treat them statistically).

Eliezer, serious question, why don't you re-brand your project as designing an Self-Improving Automated Science Machine rather than a Seed AI and generalize Friendly AI to Friendly Optimization (or similar)? It seems to me that: (a) this would be more accurate since it's not obvious (to me at least) that individual humans straightforwardly exhibit the traits you describe as "intelligence"; and (b) you'd avoid 90% of the criticism directed at you. You could, for example, avoid the usual "people have been promising AI for 60 years" line of argument.