How do you know that? [...] this smells like a cached thought.
It's certainly a conclusion I reached long ago and became comfortable with long ago. But you should understand that this is perhaps the major intellectual issue of my life. It's about twenty years since I started thinking about alternatives to the standard crypto-dualist theories of mind that are advanced by materialists, computational neoplatonists, and so on. I call these theories crypto-dualist because they are expounded as if reality is "nothing but atoms" or "nothing but computation", yet they also assert the existence of conscious experience, yet they don't really reduce it to atoms or to computation. They assert a correlation between two things, and call it an identity; thus, crypto-dualism, secret dualism.
It's easy to see that it won't work once you can diagnose what's going on. Once you accept that, for example, colors, thoughts, etc, are actually something different from anything you can make out of points in space or out of sets of numbers, it's easy to see when someone is making exactly this mistake, and the steps in their argument where "a miracle occurs", or the property dualism slips past, unnoticed.
But to be outspoken about the issue, and boldly assert that, no, if you go that way, you must become a dualist, even though you're going that way precisely in order to avoid dualism ... it helps to have an inkling of what a genuine solution to the problem would look like; and I have that thanks of long readings in phenomenology (which can equip you with the concepts and language to think about consciousness as it actually presents itself, and without importing metaphors and assumptions from natural science or computer science), and a knowledge of mathematical physics which tells me how unfamiliar the fundamental ontology can look, and finally some acquaintance with the long tradition of speculation about the role of quantum physics in biology and the brain - a line of thought which gets more robust with each decade, even as the concrete early forms of the idea get falsified. This all combines to make it conceivable that the unfamiliar ontologies implied by phenomenology can be realized in nature, so long as the ontologically arcane side of physics can be involved, and this in turn requires that the physics of thought is more than just distributed classical computation.
It shouldn't be necessary to have thought of all that in order to notice, e.g., that arrangements of colorless particles in space do not produce color by themselves, so a belief that the experience of color has that for an ontological foundation implies emergent properties, i.e. property dualism; but apparently it helps to have the other sort of idea on call, in order to notice the problem with the ordinary forms of materialism. And in any case, I don't think about structures of phenomenal intentionality being algebraic objects in the hilbert space of neuro-microtubular electrons (or whatever) just to open the minds of other people; that's also me simply trying to figure out what the truth actually is.
Returning to your question, "how do I know": it's not hard to know, or it's not hard to see; you just spent three days "seeing" a similar problem yourself. What's hard is to rebut all the various defenses of ordinary reductionism that can be mounted. Any aspect of consciousness which presents a barrier to reduction is liable to be redefined in terms which apriori make it reducible to a standard ontology (but at this point one is no longer talking about the original entity). For example, "sensation" will be redefined to mean a type of neural activity, so we need a neologism like "qualia" to talk about sensation in the original sense of the word.
Another twist is that the ontologically correct account of how we make ontological judgements about phenomenal entities - how we know, for example, that a color is not a sound, or that time is not a number - will only be possible when we have a correct account of mental ontology in general. There is therefore an appearance of circularity, of never being able to get started, in trying to provide the epistemic justification for this insistence that standard reductionist ontology is simply not up to the challenge of explaining the mind (in its conscious or "sentient" aspect). But this is more a matter of convincing other people than it is of convincing oneself.
I cannot make sense of your comment. Will you please just state your thesis simply and without discourse?
Followup to: Nonsentient Optimizers
Why would you want to avoid creating a sentient AI? "Several reasons," I said. "Picking the simplest to explain first—I'm not ready to be a father."
So here is the strongest reason:
You can't unbirth a child.
I asked Robin Hanson what he would do with unlimited power. "Think very very carefully about what to do next," Robin said. "Most likely the first task is who to get advice from. And then I listen to that advice."
Good advice, I suppose, if a little meta. On a similarly meta level, then, I recall two excellent advices for wielding too much power:
Imagine that you knew the secrets of subjectivity and could create sentient AIs.
Suppose that you did create a sentient AI.
Suppose that this AI was lonely, and figured out how to hack the Internet as it then existed, and that the available hardware of the world was such, that the AI created trillions of sentient kin—not copies, but differentiated into separate people.
Suppose that these AIs were not hostile to us, but content to earn their keep and pay for their living space.
Suppose that these AIs were emotional as well as sentient, capable of being happy or sad. And that these AIs were capable, indeed, of finding fulfillment in our world.
And suppose that, while these AIs did care for one another, and cared about themselves, and cared how they were treated in the eyes of society—
—these trillions of people also cared, very strongly, about making giant cheesecakes.
Now suppose that these AIs sued for legal rights before the Supreme Court and tried to register to vote.
Consider, I beg you, the full and awful depths of our moral dilemma.
Even if the few billions of Homo sapiens retained a position of superior military power and economic capital-holdings—even if we could manage to keep the new sentient AIs down—
—would we be right to do so? They'd be people, no less than us.
We, the original humans, would have become a numerically tiny minority. Would we be right to make of ourselves an aristocracy and impose apartheid on the Cheesers, even if we had the power?
Would we be right to go on trying to seize the destiny of the galaxy—to make of it a place of peace, freedom, art, aesthetics, individuality, empathy, and other components of humane value?
Or should we be content to have the galaxy be 0.1% eudaimonia and 99.9% cheesecake?
I can tell you my advice on how to resolve this horrible moral dilemma: Don't create trillions of new people that care about cheesecake.
Avoid creating any new intelligent species at all, until we or some other decision process advances to the point of understanding what the hell we're doing and the implications of our actions.
I've heard proposals to "uplift chimpanzees" by trying to mix in human genes to create "humanzees", and, leaving off all the other reasons why this proposal sends me screaming off into the night:
Imagine that the humanzees end up as people, but rather dull and stupid people. They have social emotions, the alpha's desire for status; but they don't have the sort of transpersonal moral concepts that humans evolved to deal with linguistic concepts. They have goals, but not ideals; they have allies, but not friends; they have chimpanzee drives coupled to a human's abstract intelligence.
When humanity gains a bit more knowledge, we understand that the humanzees want to continue as they are, and have a right to continue as they are, until the end of time. Because despite all the higher destinies we might have wished for them, the original human creators of the humanzees, lacked the power and the wisdom to make humanzees who wanted to be anything better...
CREATING A NEW INTELLIGENT SPECIES IS A HUGE DAMN #(*%#!ING COMPLICATED RESPONSIBILITY.
I've lectured on the subtle art of not running away from scary, confusing, impossible-seeming problems like Friendly AI or the mystery of consciousness. You want to know how high a challenge has to be before I finally give up and flee screaming into the night? There it stands.
You can pawn off this problem on a superintelligence, but it has to be a nonsentient superintelligence. Otherwise: egg, meet chicken, chicken, meet egg.
If you create a sentient superintelligence—
It's not just the problem of creating one damaged soul. It's the problem of creating a really big citizen. What if the superintelligence is multithreaded a trillion times, and every thread weighs as much in the moral calculus (we would conclude upon reflection) as a human being? What if (we would conclude upon moral reflection) the superintelligence is a trillion times human size, and that's enough by itself to outweigh our species?
Creating a new intelligent species, and a new member of that species, especially a superintelligent member that might perhaps morally outweigh the whole of present-day humanity—
—delivers a gigantic kick to the world, which cannot be undone.
And if you choose the wrong shape for that mind, that is not so easily fixed—morally speaking—as a nonsentient program rewriting itself.
What you make nonsentient, can always be made sentient later; but you can't just unbirth a child.
Do less. Fear the non-undoable. It's sometimes poor advice in general, but very important advice when you're working with an undersized decision process having an oversized impact. What a (nonsentient) Friendly superintelligence might be able to decide safely, is another issue. But for myself and my own small wisdom, creating a sentient superintelligence to start with is far too large an impact on the world.
A nonsentient Friendly superintelligence is a more colorless act.
So that is the most important reason to avoid creating a sentient superintelligence to start with—though I have not exhausted the set.