You can have some fun with people whose anticipations get out of sync with what they believe they believe.
I was once at a dinner party, trying to explain to a man what I did for a living, when he said: "I don't believe Artificial Intelligence is possible because only God can make a soul."
At this point I must have been divinely inspired, because I instantly responded: "You mean if I can make an Artificial Intelligence, it proves your religion is false?"
He said, "What?"
I said, "Well, if your religion predicts that I can't possibly make an Artificial Intelligence, then, if I make an Artificial Intelligence, it means your religion is false. Either your religion allows that it might be possible for me to build an AI; or, if I build an AI, that disproves your religion."
There was a pause, as the one realized he had just made his hypothesis vulnerable to falsification, and then he said, "Well, I didn't mean that you couldn't make an intelligence, just that it couldn't be emotional in the same way we are."
I said, "So if I make an Artificial Intelligence that, without being deliberately preprogrammed with any sort of script, starts talking about an emotional life that sounds like ours, that means your religion is wrong."
He said, "Well, um, I guess we may have to agree to disagree on this."
I said: "No, we can't, actually. There's a theorem of rationality called Aumann's Agreement Theorem which shows that no two rationalists can agree to disagree. If two people disagree with each other, at least one of them must be doing something wrong."
We went back and forth on this briefly. Finally, he said, "Well, I guess I was really trying to say that I don't think you can make something eternal."
I said, "Well, I don't think so either! I'm glad we were able to reach agreement on this, as Aumann's Agreement Theorem requires." I stretched out my hand, and he shook it, and then he wandered away.
A woman who had stood nearby, listening to the conversation, said to me gravely, "That was beautiful."
"Thank you very much," I said.
Part of the sequence Mysterious Answers to Mysterious Questions
Next post: "Professing and Cheering"
Previous post: "Belief in Belief"
Douglas, (1) what makes you think that anyone was suggesting that "AI" and "soul" have the same meaning?, (2) in what way would "an AI is not a soul" be a useful substitute for anything else said in this discussion?, and (3) why should comparing the two notions lead to any problems, and in particular to whatever you're calling "the problem" here?
I don't think it's any more obvious that there are no AIs than that there are no souls. That is: perhaps, despite appearances, my computer is really intelligent, and thinks in some manner quite different from the computational processes I know it performs, but which is none the less somehow based on them. There is just as much evidence for this (admittedly bizarre) hypothesis as there is for the existence of souls.
(On some side issues: I think "God" should be capitalized when it's being used as a proper name, and not otherwise. Thus, "the Christian god" but "May God bless you" or "Oh God, what a stupid idea". Note that this has nothing to do with, e.g., whether one believes in the existence of any such being or one's opinions about whether he/she/it does/would deserve respect. I don't understand why "eternal" should be part of the definition of "soul". The point of Aumann's theorem is that observing someone else's opinion and how it changes gives you information about what the other person knows that you don't.)