No-one's telling me that a heap of sand has an "inside". It's a fuzzy concept and the fuzziness doesn't cause any problems because it's just a loose categorization. But the individual consciousness actually exists and is actually distinct from things that aren't it, so in a physical ontology it has to correspond to a hard-edged concept.
Degree-of-existence seems likely to be well-defined and useful, and may play a part in, for example, quantum mechanics.
However, my new response to your argument is that, if you're not denying current physics, but just ontologically reorganizing it., then you're vulnerable to the same objection. You can declare something to be Ontologically Fundamental, but it will still mathematically be a heap of sand, and you can still physically remove a grain. We're all in the same boat.
Consider Cyc. Isn't one of the problems of Cyc that it can't distinguish itself from the world? It can distinguish the Cyc-symbol from other symbols, but only in the same way that it distinguishes any symbol from any other symbol. Any attempt to make it treat the Cyc-symbol really differently requires that the Cyc-symbol gets special treatment on the algorithmic level.
Do you think Cyc could not be programmed to treat itself different from others without use of a quantum computer? If not, how can you make inferences about quantum entanglement from facts about our programming.
Does Cyc have sensors or something? If it does/did, it seems like it would algorithmically treat raw sensory data as separate from symbols and world-models.
In other words, so long as we talk simply about computation, there is nothing at all to inherently make an AI insist that its "experience" can't be made of physical entities. It's just a matter of ontological presuppositions.
Is there anything to inherently prevent it from insisting that? Should we accept our ontological presuppositions at face value?
I would be able to tell the difference between an ontology in which they exist, and an ontology in which they don't.
No you wouldn't. People can't tell the difference between ontologies any more then math changes if you print its theorems in a different color. People can tell the difference between different mathematical laws of physics, or different arrangements of stuff within those laws. What you notice is that you have a specific class of gensyms that can't have relations of reduction for other symbols, or something else computational. Facts about ontology are totally orthogonal to facts about things that influence what words you type.
A neuron is a glob of trillions of atoms doing inconceivably many things at once. You're focusing on a few of the simple differential sub-perceptions which make up the experience of looking at that image, associating them in your mind with certain gross changes of state in that glob of atoms, and proposing that the experience is identical to a set of several such simultaneous changes occurring in a few neurons. In doing so, you're neglecting both the bulk of the physical events occurring elsewhere in the neurons, and the fundamental dissimilarity between "staring at a few homogeneous patches of color" and "billions of ions cascading through a membrane".
My consciousness is a computation based mainly or entirely on regularities the size of a single neuron or bigger, much like the browser I'm typing in is based on regularities the size of a transistor. I wouldn't expect to notice if my images were, really, fundamentally, completely different. I wouldn't expect to notice if something physical happened - the number of ions was cut by a factor of a million and made the opposite charge, but it the functions from impulses to impulses computed by neurons were the same.
It's more like the difference between night and day. It is possible to attain a higher perspective which unifies them, but you don't get there by saying that day is just night by another name.
Uniform color and edgeness are as different as night and day.
(part 2 of reply)
In other words, so long as we talk simply about computation, there is nothing at all to inherently make an AI insist that its "experience" can't be made of physical entities. It's just a matter of ontological presuppositions.
Is there anything to inherently prevent it from insisting that? Should we accept our ontological presuppositions at face value?
See next section.
...I would be able to tell the difference between an ontology in which they exist, and an ontology in which they don't.
No you wouldn't. People can't tell th
...at least not if you accept a certain line of anthropic argument.
Thomas Nagel famously challenged the philosophical world to come to terms with qualia in his essay "What is it Like to Be a Bat?". Bats, with sensory systems so completely different from those of humans, must have exotic bat qualia that we could never imagine. Even if we deduce all the physical principles behind echolocation, even if we could specify the movement of every atom in a bat's senses and nervous system that represents its knowledge of where an echolocated insect is, we still have no idea what it's like to feel a subjective echolocation quale.
Anthropic reasoning is the idea that you can reason conditioning on your own existence. For example, the Doomsday Argument says that you would be more likely to exist in the present day if the overall number of future humans was medium-sized instead of humongous, therefore since you exist in the present day, there must be only a medium-sized number of future humans, and the apocalypse must be nigh, for values of nigh equal to "within a few hundred years or so".
The Buddhists have a parable to motivate young seekers after enlightenment. They say - there are zillions upon zillions of insects, trillions upon trillions of lesser animals, and only a relative handful of human beings. For a reincarnating soul to be born as a human being, then, is a rare and precious gift, and an opportunity that should be seized with great enthusiasm, as it will be endless eons before it comes around again.
Whatever one thinks of reincarnation, the parable raises an interesting point. Considering the vast number of non-human animals compared to humans, the probability of being a human is vanishingly low. Therefore, chances are that if I could be an animal, I would be. This makes a strong anthropic argument that it is impossible for me to be an animal.
The phrase "for me to be an animal" may sound nonsensical, but "why am I me, rather than an animal?" is not obviously sillier than "why am I me, rather than a person from the far future?". If the doomsday argument is sufficient to prove that some catastrophe is preventing me from being one of a trillion spacefaring citizens of the colonized galaxy, this argument hints that something is preventing me from being one of a trillion bats or birds or insects.
And this could be that animals lack subjective experience. This would explain quite nicely why I'm not an animal: because you can't be an animal, any more than you can be a toaster. So Thomas Nagel can stop worrying about what it's like to be a bat, and the rest of us can eat veal and foie gras guilt-free.
But before we break out the dolphin sausages - this is a pretty weird conclusion. It suggests there's a qualitative and discontinuous difference between the nervous system of other beings and our own, not just in what capacities they have but in the way they cause experience. It should make dualists a little bit happier and materialists a little bit more confused (though it's far from knockout proof of either).
The most significant objection I can think of is that it is significant not that we are beings with experiences, but that we know we are beings with experiences and can self-identify as conscious - a distinction that applies only to humans and maybe to some species like apes and dolphins who are rare enough not to throw off the numbers. But why can't we use the reference class of conscious beings if we want to? One might as well consider it significant only that we are beings who make anthropic arguments, and imagine there will be no Doomsday but that anthropic reasoning will fall out of favor in a few decades.
But I still don't fully accept this argument, and I'd be pretty happy if someone could find a more substantial flaw in it.