(This post grew out of an old conversation with Wei Dai.)
Imagine a person sitting in a room, communicating with the outside world through a terminal. Further imagine that the person knows some secret fact (e.g. that the Moon landings were a hoax), but is absolutely committed to never revealing their knowledge of it in any way.
Can you, by observing the input-output behavior of the system, distinguish it from a person who doesn't know the secret, or knows some other secret instead?
Clearly the only reasonable answer is "no, not in general".
Now imagine a person in the same situation, claiming to possess some mental skill that's hard for you to verify (e.g. visualizing four-dimensional objects in their mind's eye). Can you, by observing the input-output behavior, distinguish it from someone who is lying about having the skill, but has a good grasp of four-dimensional math otherwise?
Again, clearly, the only reasonable answer is "not in general".
Now imagine a sealed box that behaves exactly like a human, dutifully saying things like "I'm conscious", "I experience red" and so on. Moreover, you know from trustworthy sources that the box was built by scanning a human brain, and then optimizing the resulting program to use less CPU and memory (preserving the same input-output behavior). Would you be willing to trust that the box is in fact conscious, and has the same internal experiences as the human brain it was created from?
A philosopher believing in computationalism would emphatically say yes. But considering the examples above, I would say I'm not sure! Not at all!
Gosh, I really don't want to start talking about morality now. But I have to point out that the "bitterness of purple" can also matter, if it features in some scheme of ideal moral reasoning. At least if you accept that this moral reasoning could require arbitrary concepts and not just ones grounded in reality.
No, I ran a deterministic procedure in my brain, called "is X well defined", on "robot pain", and it returned "no". It's only subjective in the sense that mine is different from yours, if you have such a procedure at all. The procedure, by the way, works by searching for alternative definitions of things, such that the given concept is neither trivial nor stupid. Unfortunately, failure to find such definitions does not produce a proof of non-existence, so I'm quite open to the idea that I missed something, it's just that you inspire little confidence.
I did not mean to imply that ideal moral reasoning is weird and unguessable....only that you should not take imperfect moral reasoning (whose?) to be the last word. The idea that deliberately causing pain is wrong is not contentious, and you don't actually have an argument against it.
That's the sense that matters.