Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.
Thagard (2012) contains a nicely compact passage on thought experiments:
Grisdale’s (2010) discussion of modern conceptions of water refutes a highly influential thought experiment that the meaning of water is largely a matter of reference to the world rather than mental representation. Putnam (1975) invited people to consider a planet, Twin Earth, that is a near duplicate of our own. The only difference is that on Twin Earth water is a more complicated substance XYZ rather than H2O. Water on Twin Earth is imagined to be indistinguishable from H2O, so people have the same mental representation of it. Nevertheless, according to Putnam, the meaning of the concept water on Twin Earth is different because it refers to XYZ rather than H2O. Putnam’s famous conclusion is that “meaning just ain’t in the head.”
The apparent conceivability of Twin Earth as identical to Earth except for the different constitution of water depends on ignorance of chemistry. As Grisdale (2010) documents, even a slight change in the chemical constitution of water produces dramatic changes in its effects. If normal hydrogen is replaced by different isotopes, deuterium or tritium, the water molecule markedly changes its chemical properties. Life would be impossible if H2O were replaced by heavy water, D2O or T2O; and compounds made of elements different from hydrogen and oxygen would be even more different in their properties. Hence Putnam’s thought experiment is scientifically incoherent: If water were not H2O, Twin Earth would not be at all like Earth. [See also Universal Fire. --Luke]
This incoherence should serve as a warning to philosophers who try to base theories on thought experiments, a practice I have criticized in relation to concepts of mind (Thagard, 2010a, ch. 2). Some philosophers have thought that the nonmaterial nature of consciousness is shown by their ability to imagine beings (zombies) who are physically just like people but who lack consciousness. It is entirely likely, however, that once the brain mechanisms that produce consciousness are better understood, it will become clear that zombies are as fanciful as Putnam’s XYZ. Just as imagining that water is XYZ is a sign only of ignorance of chemistry, imagining that consciousness is nonbiological may well turn out to reveal ignorance rather than some profound conceptual truth about the nature of mind. Of course, the hypothesis that consciousness is a brain process is not part of most people’s everyday concept of consciousness, but psychological concepts can progress just like ones in physics and chemistry. [See also the Zombies Sequence. --Luke]
(1) Calculate which action A would maximize parameter P, based on existing data set D. (2) Summarize this calculation in a user-friendly manner, including what Action A is, what likely intermediate outcomes it would cause, what other actions would result in high values of P, etc.
In contrast, an agent AI would:
(1) Calculate which action, A, would maximize parameter P, based on existing data set D. (2) Execute Action A.
The idea being that an AI, asked to "prevent human suffering", would come up with two plans:
- Kill all human.
- Cure all diseases, make everyone young and immortal.
Then the agent AI would go out and kill everyone, while the tool AI would give us the list and we would pick the second one. In the following, I'll assume the AI is superintelligent, and has no other objectives than what we give it.
Annoyance wants us to stop talking about fancy techniques and get back to basics. I disagree with the philosophy behind his statement, but the principle is sound. In many areas of life - I'm thinking mostly of sports, but not for lack of alternatives - mastery of the basics beats poorly-grounded fancy techniques every time.
One basic of rationality is paying close attention to an argument. Dissecting it to avoid rhetorical tricks, hidden fallacies, and other Dark Arts. I've been working on this for years, and I still fall short on a regular basis.
Medical educators have started emphasizing case studies in their curricula. Instead of studying arcane principles of disease, student doctors cooperate to analyze a particular patient in detail, ennumerate the principles needed to diagnose her illness, and pay special attention to any errors the patients' doctors made during the treatment. The cases may be rare tropical infections, but they're more often the same everyday diseases common in the general population, forcing the student doctors to always keep the basics in mind. We could do with a tradition of case studies in rationality, though we'd need safeguards to prevent degeneration into political discussion.
Case studies in medicine are most interesting when all the student doctors disagree with each other. To that end, I've chosen as the first case a statement that received sixteen upvotes on Less Wrong, maybe the highest I've ever seen for a comment. I don't mean to insult or embarass everyone who liked it. I liked it too. My cursor was already hovering above the "Vote Up" button by the time I starting having second thoughts. But it deserves dissection, and its popularity gives me a ready response when someone says this material is too basic for 'master rationalists' like ourselves:
In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment.
This anecdote is short, witty, flattering, and utterly opaque to reason. It bears all the hallmarks of the Dark Arts.
Baboons... literally have been the textbook example of a highly aggressive, male-dominated, hierarchical society. Because these animals hunt, because they live in these aggressive troupes on the Savannah... they have a constant baseline level of aggression which inevitably spills over into their social lives.
Scientists have never observed a baboon troupe that wasn't highly aggressive, and they have compelling reasons to think this is simply baboon nature, written into their genes. Inescapable.
Or at least, that was true until the 1980s, when Kenya experienced a tourism boom.
Sapolsky was a grad student, studying his first baboon troupe. A new tourist lodge was built at the edge of the forest where his baboons lived. The owners of the lodge dug a hole behind the lodge and dumped their trash there every morning, after which the males of several baboon troupes — including Sapolsky's — would fight over this pungent bounty.
Before too long, someone noticed the baboons didn't look too good. It turned out they had eaten some infected meat and developed tuberculosis, which kills baboons in weeks. Their hands rotted away, so they hobbled around on their elbows. Half the males in Sapolsky's troupe died.
This had a surprising effect. There was now almost no violence in the troupe. Males often reciprocated when females groomed them, and males even groomed other males. To a baboonologist, this was like watching Mike Tyson suddenly stop swinging in a heavyweight fight to start nuzzling Evander Holyfield. It never happened.