Readers of Less Wrong may be interested in this New Scientist article by Noel Sharkey, titled Why AI is a dangerous dream, in which he attacks Kurzweil's and Moravec's "fairy tale" predictions and questions whether intelligence is computational ("[the mind] could be a physical system that cannot be recreated by a computer").

[edit] I thought this would go without saying, but I suspect the downvotes speak otherwise, so: I strongly disagree with the content of this article. I still consider it interesting because it is useful to be aware of differing and potentially popular perspectives on these subjects (and Sharkey is something of a "populist" scientist). I think the opinions it espouses are staggeringly ill-conceived, however.

Two comments. First, your point about counterfactuals is very valid. Hofstadter wrote an essay about how we tend to automatically only consider certain counterfactuals, when an infinite variety are theoretically possible. There are many ways that the world might be changed so that Joe one-boxes. A crack in the earth might open and swallow one box, allowing Joe to take only the other. Someone might have offered Joe a billion dollars to take one box. Joe might aim to take two but suffer a neurological spasm which caused him to grasp only one box and then leave. And so on. Counterfactuals are a weak and uncertain tool.

My second point is with regard to determinism. What if the word in general, and Joe in particular, is nondeterministic? What if QM is true but the MWI is not, or some other form of nondeterminism prevails? Ideally, you should not base your analysis on the assumption of determinism.

Two comments. First, your point about counterfactuals is very valid. Hofstadter wrote an essay about how we tend to automatically only consider certain counterfactuals, when an infinite variety are theoretically possible. There are many ways that the world might be changed so that Joe one-boxes. A crack in the earth might open and swallow one box, allowing Joe to take only the other. Someone might have offered Joe a billion dollars to take one box. Joe might aim to take two but suffer a neurological spasm which caused him to grasp only one box and then leave. And so on. Counterfactuals are a weak and uncertain tool.

My second point is with regard to determinism. What if the word in general, and Joe in particular, is nondeterministic? What if QM is true but the MWI is not, or some other form of nondeterminism prevails? Ideally, you should not base your analysis on the assumption of determinism.

Two comments. First, your point about counterfactuals is very valid. Hofstadter wrote an essay about how we tend to automatically only consider certain counterfactuals, when an infinite variety are theoretically possible. There are many ways that the world might be changed so that Joe one-boxes. A crack in the earth might open and swallow one box, allowing Joe to take only the other. Someone might have offered Joe a billion dollars to take one box. Joe might aim to take two but suffer a neurological spasm which caused him to grasp only one box and then leave. And so on. Counterfactuals are a weak and uncertain tool.

My second point is with regard to determinism. What if the word in general, and Joe in particular, is nondeterministic? What if QM is true but the MWI is not, or some other form of nondeterminism prevails? Ideally, you should not base your analysis on the assumption of determinism.

Two comments. First, your point about counterfactuals is very valid. Hofstadter wrote an essay about how we tend to automatically only consider certain counterfactuals, when an infinite variety are theoretically possible. There are many ways that the world might be changed so that Joe one-boxes. A crack in the earth might open and swallow one box, allowing Joe to take only the other. Someone might have offered Joe a billion dollars to take one box. Joe might aim to take two but suffer a neurological spasm which caused him to grasp only one box and then leave. And so on. Counterfactuals are a weak and uncertain tool.

My second point is with regard to determinism. What if the word in general, and Joe in particular, is nondeterministic? What if QM is true but the MWI is not, or some other form of nondeterminism prevails? Ideally, you should not base your analysis on the assumption of determinism.

We talk a lot here about creating Artificial Intelligence. What I think Tiiba is asking about is how we might create Artificial Consciousness, or Artificial Sentience. Could there be a being which is conscious and which can suffer and have other experiences, but which is not intelligent? Contrariwise, could there be a being which is intelligent and a great problem solver, able to act as a Bayesian agent very effectively and achieve goals, but which is not conscious, not sentient, has no qualia, cannot be said to suffer? Are these two properties, intelligence and consciousness, independent or intrinsically linked?

Acknowledging the limited value of introspection, nevertheless I can remember times which I was close to experiencing "pure consciousness", with no conscious problem-solving activity at all. Perhaps I was entranced by a beautiful sunset, or haunting musical performance. My whole being seemed to be pure experience, pure consciousness, with no particular need for intelligence, Bayesian optimization, goal satisfaction, or any of the other paraphernalia which we associate with intelligence. This suggests to me that it is at least plausible that consciousness does not require intelligence.

In the other direction, the idea of an intelligence problem solver devoid of consciousness is an element in many powerful, fictional dystopias. Even Eliezer's paperclip maximizer partakes of this trope. It seems that we have little difficulty imagining intelligence without consciousness, without awareness, sentience, qualia, the ability to suffer.

If we provisionally assume that the two qualities are independent, it raises the question of how we might program consciousness (even if we only want to know how, to avoid doing it accidentally). is it possible that even relatively simple programs may be conscious, may be capable of feeling real pain and suffering, as well as pleasure and joy? Is there any kind of research program that could shed light on these questions?

Reading the comments here, there seem to be two issues entangled. One is which organisms are capable of suffering (which is probably roughly the same set that is capable of experiencing qualia; we might call this the set of sentient beings). The other is which entities we would care about and perhaps try to help.

I don't think the second question is really relevant here. It is not the issue Tiiba is trying to raise. If you're a selfish bastard, or a saintly altruist, fine. That doesn't matter. What matters is what constitutes a sentient being which can experience suffering and similar sensations.

Let us try to devote our attention to this question, and not the issue of what our personal policies are towards helping other people.

On the other hand, for all we know, since the laws of physics in this universe allow for magic, the spell might actually do what the Council thinks it does - summons a hero who brings along the proper kind of luck for getting through the current crisis.

"I summon Deus Ex Machina!"

I know what Eliezer intended the story to mean, but narrative causality seems like a more likely culprit than the anthropic principle for this particular world's survival. Considering this is a world in which the events of Lord of the Rings actually happened, if I were the hero, I'd be assuming that there's a writer of some kind involved.

I remember reading once about an experiment that was said to make rats superstitious.

These rats were used in learning experiments. They would be put into a special cage and they'd have to do something to get a treat. Maybe they'd have to push a lever, or go to a certain spot. But they were pretty good at learning whatever they had to do. They were smart rats. They knew the score, they knew what the cage was for.

So they did a new experiment, where they put them into the training cage as usual. But instead of what they did bringing the treat, they always got a treat exactly 30 seconds after going into the cage. This continued for a while, and what happened was the rats each learned an individual behavior to bring the treat. One would go to a corner, another would turn in circles, another would stand up on its hind feet. And sure enough, the treat came. Their trick worked.

I imagine the society in Eliezer's story had something similar happen. Given the anthropic effect we are postulating, they don't actually have to do anything - a certain fraction of the worlds will get lucky and survive. But after it happens a few times, the survivors may well assume that what they were doing at the time their "luck" arrived was causative. In this case, they had a hero who seemed to get lucky. Maybe several heroes. And then somewhere they got the idea of summoning them from other worlds. After all, if they need a lucky hero to save them, they should get the luckiest heroes they can find. (I wonder what this hero had done to earn their selection?)

But there would be just as many other worlds, even instances of the exact same world, which have developed their own superstitions about what defeats the evil. They each carry out their rituals, and in each case, it works - for the survivors. We just choose to eavesdrop on a world which had a particularly interesting and amusing superstition.

"Ah..." Ghufhus said, puzzled. "How do you... know all this? Is there a Counter-Force in your own world?"

Fool, Aerhien thought to herself. The hero had seemed puzzled by the idea, at first, and had needed to ask for examples. She decided then and there that Ghufhus would meet with an accident before the next council meeting; their world had no room for stupid Eerionnath.

That's a bit harsh, don't you think? Maybe the Counter-Force is known by a different name in the Hero's own world, and he asked for examples to make sure they're the same thing.

Actually, why doesn't the Hero's world have a Counter-Force? Shouldn't every world have something like it? How many times have our world escaped from the brink of nuclear annihilation, for example?