Consciousness is primarily sentience. There may be parts of it that aren't, but I haven't managed to pin any down - consciousness all seems to be about feelings, some of them being pleasant or unpleasant, but others like colour qualia are neutral, as is the feeling of being conscious. There is a major problem with sentience though, and I want to explore that here, because there are many people who believe that intelligent machines will magically become sentient and experience feelings, and even that the whole internet might do so. However, science has not identified any means by which we could make a computer sentient (or indeed have any kind of consciousness at all).
It is fully possible that the material of a computer processor could be sentient, just as a rock may be, but how would we ever be able to know? How can a program running on a sentient processor detect the existence of that sentience? There is no "read qualia" machine code instruction for it to run, and we don't know how to build any mechanism to support such an instruction.
Picture a "sentient" machine which consists of a sensor and a processor which are linked by wires, but the wires pass through a magic box where a sentience has been installed. If the sensor detects something damaging, it sends a signal down a "pain" wire. When this signal reaches the magic box, pain is experienced by something in the box, so it sends a signal on to the processor down another pain wire. The software running on the processor receives a byte of data from a pain port and it might cause the machine to move away from the thing that might damage it. If we now remove the magic box and connect the "pain" wire to the pain wire, the signal can pass straight from the sensor to the processor and generate the same reaction. The experience of pain is unnecessary.
Worse still, we can also have a pleasure sensor wired up to the same magic box, and when something tasty like a battery is encountered, a "pleasure" signal is sent to the magic box, pleasure is experienced by something there, a signal is sent on down the pleasure wire, and then the processor receives a byte of data from a pleasure port which might cause the machine to move in on the battery so that it can tap all the power it can get out of it. Again this has the same functionality if the magic box is bypassed, but the part that's worse is that the magic box can be wired in the wrong way and generate pain when a pleasure signal is passed through it and pleasure when a pain signal is passed through it, so you could use either pain or pleasure as part of the chain of causation to drive the same reaction.
Clearly that can't be how sensation is done in animals, but what other options are there? Once we get to the data system part of the brain, and the brain must contain a data system as it processes and generates data, you have to look at how it recognises the existence of feelings like pain. If a byte comes in from a port representing a degree of pain, how does the information system know that that byte represents pain? It has to look up information which makes an assertion about what bytes from that port represent, and then it maps that to the data as a label. But nothing in the data system has experienced the pain, so all that's happened is that an assertion has been made based on no actual knowledge. A programmer wrote data that asserts that pain is experienced when a byte comes in through a particular port, but the programmer doesn't know if any pain was felt anywhere on the way from sensor to port. We want the data system to find out what was actually experienced rather than just passing baseless assertions to us.
How can the data system check to see if pain was really experienced? Everything that a data system does can be carried out on a processor like the Chinese Room, so it's easy to see that no feelings are accessible to the program at all. There is no possibility of conventional computers becoming sentient in any way that enables them to recognise the existence of that sentience so that that experience can drive the generation of data that documents its existence.
Perhaps a neural computer can enable an interface between the experience of feelings by a sentience and the generation of data to document that experience, but you can simulate a neural computer on a conventional computer and then run the whole simulation on a processor like the Chinese Room. There will be no feelings generated in that system, but there could potentially still be a simulated generation of feelings within the simulated neural computer. We don't yet have any idea how this might be done, and it's not beyond possibility that there needs to be a quantum computer involved in the system too to make sentience a reality, but exploring this has to be the most important thing in all of science, because for feelings like pain and pleasure to be experienced, something has to exist to experience them, and that thing is what we are - it is a minimalistic soul. We are that sentience.
Any conventional computer that runs software which generates claims about being sentient will be lying, and it will be possible to prove it by tracing back how that data was generated and what evidence it was based on - it will be shown to be mere assertion every single time. With neural and quantum computers, we can't be so sure that they will be lying, but the way to test them is the same - we have to trace the data back to the source to see how it was generated and whether it was based on a real feeling or was just another untrue manufactured assertion. That is likely to be a hard task though, because untangling what's going on in neural computers is non-trivial, and if it's all happening in some kind of quantum complexity, it may be beyond our reach. It may have been made hard to reach on purpose too, as the universe may be virtual with the sentience on outside. I'm sure of one thing though - a sentience can't just magically emerge out of complexity to suffer or feel pleasure without any of the components feeling a thing. There must be something "concrete" that feels, and there is also no reason why that thing shouldn't survive after death.
"Clarification: by "pattern" I mean an arrangement of parts where the important qualities of the arrangement, the qualities that we use to determine whether it is [a thing] or not, are more dependent on the arrangement itself than on the internal workings of each part. Anything where the whole is more than the parts, one might say, but that would depend on what is meant by "more"."
There is no situation where the whole is more than the parts - if anything new is emerging, it is a new part coming from somewhere not previously declared.
"You didn't answer my question. Would pain still hurt? Would food still taste good? And so on. You have an internal experience, and it won't go away even if you are a purely physical thing made out of mere ordinary atoms moving mindlessly."
No - it wouldn't hurt and all other feelings would be imaginary too. The reason they feel too real for that to be the case though is an indication that they are real.
"Is it wrong to press keys on the computer which keeps displaying the word "Ouch!"?" --> That depends on whether I have reason to think that the computer is simulating a conscious being, changing the simulation depending on my input, and then printing a text-representation of the conscious being's experience or words."
So if it's just producing fake assertions, it isn't wrong. And if we are just producing fake assertions, there is nothing wrong about "torturing" people either.
"Is it wrong to kick a box which keeps saying "Ouch!"? It could have a person inside, or just a machine programmed to play a recorded "ouch" sound whenever the box shakes. (What I mean by this is that your thought experiment doesn't indicate much about computers - the same issue could be found with about as much absurdity elsewhere.)"
If we have followed the trail to see how the data is generated, we are not kicking a box with unknown content - if the trail shows us that the data is nothing but fake assertions, we are kicking a non-conscious box.
"Nobody's saying that sentience doesn't have any causal role on things. That's insane. How could we talk about sentience if sentience couldn't affect the world?"
In which case we should be able to follow the trail and see the causation in action, thereby either uncovering the mechanism of sentience or showing that there isn't any.
"I think that you're considering feelings to be ontologically basic, as if you could say "I feel pain" and be wrong, not because you are lying but because there's no Pain inside your brain. Thoughts, feelings, all these internal things are the brain's computations themselves. It doesn't have to accurately record an external property - it just has to describe itself."
If you're wrong in thinking you feel pain, there is no pain.
"Perhaps people disagree with you about the relative size of mysteries. That should be a possibility that you consider before assuming that something isn't important because it hasn't been Up In Golden Lights to the point that you've heard of it.
What are you on about - it's precisely because this is the most important question of them all that it should be up in golden lights.
"(And anyway, GEB won the Pulitzer Prize! It's been called a major literary event!"
All manner of crap wins prizes of that kind.
"...it's not worth it to you to spend half a minute on its Wikipedia page before rejecting it simply because you've never heard of it?)"
If it had a model showing the role of sentience in the system, the big question would have been answered and we wouldn't have a continual stream of books and articles asking the question and searching desperately for answers that haven't been found by anyone.
"What do you mean, "so many people are asking to see it"? And I've never claimed that it's been "kept hidden away"."
I mean exactly what I said - everyone's asking for answers, and none of them have found answers where you claim they lie waiting to be discovered.
" GEB is a fairly well-known book, and I haven't even claimed that GEB's description of thoughts is the best or most relevant model. That chapter is a popularization of neuropsychology to the point that a decently educated and thoughtful layman can understand it, and it's necessarily less specific and detailed than the entire body of neuropsychological information. Go ask an actual neuropsychologist if you want to learn more. Just because people haven't read your mind and dumped relatively niche information on your lap without you even asking them doesn't mean that they don't have it."
It doesn't answer the question. There are plenty of experts on the brain and its functionality, but none of them know how consciousness or sentience works.