Thagard (2012) contains a nicely compact passage on thought experiments:
Grisdale’s (2010) discussion of modern conceptions of water refutes a highly influential thought experiment that the meaning of water is largely a matter of reference to the world rather than mental representation. Putnam (1975) invited people to consider a planet, Twin Earth, that is a near duplicate of our own. The only difference is that on Twin Earth water is a more complicated substance XYZ rather than H2O. Water on Twin Earth is imagined to be indistinguishable from H2O, so people have the same mental representation of it. Nevertheless, according to Putnam, the meaning of the concept water on Twin Earth is different because it refers to XYZ rather than H2O. Putnam’s famous conclusion is that “meaning just ain’t in the head.”
The apparent conceivability of Twin Earth as identical to Earth except for the different constitution of water depends on ignorance of chemistry. As Grisdale (2010) documents, even a slight change in the chemical constitution of water produces dramatic changes in its effects. If normal hydrogen is replaced by different isotopes, deuterium or tritium, the water molecule markedly changes its chemical properties. Life would be impossible if H2O were replaced by heavy water, D2O or T2O; and compounds made of elements different from hydrogen and oxygen would be even more different in their properties. Hence Putnam’s thought experiment is scientifically incoherent: If water were not H2O, Twin Earth would not be at all like Earth. [See also Universal Fire. --Luke]
This incoherence should serve as a warning to philosophers who try to base theories on thought experiments, a practice I have criticized in relation to concepts of mind (Thagard, 2010a, ch. 2). Some philosophers have thought that the nonmaterial nature of consciousness is shown by their ability to imagine beings (zombies) who are physically just like people but who lack consciousness. It is entirely likely, however, that once the brain mechanisms that produce consciousness are better understood, it will become clear that zombies are as fanciful as Putnam’s XYZ. Just as imagining that water is XYZ is a sign only of ignorance of chemistry, imagining that consciousness is nonbiological may well turn out to reveal ignorance rather than some profound conceptual truth about the nature of mind. Of course, the hypothesis that consciousness is a brain process is not part of most people’s everyday concept of consciousness, but psychological concepts can progress just like ones in physics and chemistry. [See also the Zombies Sequence. --Luke]
I am most assuredly fighting the hypothetical (I'm familiar with and disagree with that link). As far as I can tell, that's what Thagard is doing too.
I'm reminded of a rebuttal to that post, about how hypotheticals are used as a trap. Putnam intentionally chose to create a scientifically incoherent world. He could have chosen a jar of acid instead of an incoherent twin-earth, but he didn't. He wanted the sort of confusion that could only come from an incoherent universe (luke links that in his quote).
I think that's Thagard's point. As he notes: these types of thought experiments are only expressions of our ignorance, and not deep insights about the mind.
I'm not quite sure why it matters that the world Putnam creates is "scientifically incoherent" - which I take to mean it conflicts with our current understanding of science?
As far as we know, the facts of science could have been different; hell, we could still be wrong about the ones we currently think we know. So our language ought to be able to cope with situations where the scientific facts are different than they actually are. It doesn't matter that Putnam's scenario can't happen in this world: it could have happened, and thinking about what ... (read more)