Thagard (2012) contains a nicely compact passage on thought experiments:
Grisdale’s (2010) discussion of modern conceptions of water refutes a highly influential thought experiment that the meaning of water is largely a matter of reference to the world rather than mental representation. Putnam (1975) invited people to consider a planet, Twin Earth, that is a near duplicate of our own. The only difference is that on Twin Earth water is a more complicated substance XYZ rather than H2O. Water on Twin Earth is imagined to be indistinguishable from H2O, so people have the same mental representation of it. Nevertheless, according to Putnam, the meaning of the concept water on Twin Earth is different because it refers to XYZ rather than H2O. Putnam’s famous conclusion is that “meaning just ain’t in the head.”
The apparent conceivability of Twin Earth as identical to Earth except for the different constitution of water depends on ignorance of chemistry. As Grisdale (2010) documents, even a slight change in the chemical constitution of water produces dramatic changes in its effects. If normal hydrogen is replaced by different isotopes, deuterium or tritium, the water molecule markedly changes its chemical properties. Life would be impossible if H2O were replaced by heavy water, D2O or T2O; and compounds made of elements different from hydrogen and oxygen would be even more different in their properties. Hence Putnam’s thought experiment is scientifically incoherent: If water were not H2O, Twin Earth would not be at all like Earth. [See also Universal Fire. --Luke]
This incoherence should serve as a warning to philosophers who try to base theories on thought experiments, a practice I have criticized in relation to concepts of mind (Thagard, 2010a, ch. 2). Some philosophers have thought that the nonmaterial nature of consciousness is shown by their ability to imagine beings (zombies) who are physically just like people but who lack consciousness. It is entirely likely, however, that once the brain mechanisms that produce consciousness are better understood, it will become clear that zombies are as fanciful as Putnam’s XYZ. Just as imagining that water is XYZ is a sign only of ignorance of chemistry, imagining that consciousness is nonbiological may well turn out to reveal ignorance rather than some profound conceptual truth about the nature of mind. Of course, the hypothesis that consciousness is a brain process is not part of most people’s everyday concept of consciousness, but psychological concepts can progress just like ones in physics and chemistry. [See also the Zombies Sequence. --Luke]
Putnam perhaps chose poor examples, but his thought-experiment works under any situation where we have limited knowledge.
Instead of Twin Earth, say that I have a jar of clear liquid on my desk. Working off of just that information (and the information that much of the clear liquid that humans keep around are water) people start calling the thing on my desk a "Jar of Water." That is, until someone knocks it over and it starts to eat through the material on my desk: obviously, that wasn't water.
Putnam doesn't think that XYZ will look like water in every circumstance: his thought-experiment includes the idea that we can distinguish between XYZ and water with, say, an electron microscope. So obviously there are some properties of XYZ that are not the same as water, or else they really would look the same under every possible circumstance.
The difference (which is where some philosophers make the mistake) is when you assume that the "thought-experiment" stuff looks like the "real" stuff in every possible circumstance. If Putnam had said that the difference between H2O and XYZ was purely ephiphenomenal or something like that, he'd be obviously wrong. For instance, if we looked at XYZ and it "fooled" us into thinking it was H2O (say, if we broke apart XYZ and got a 2:1 ration of oxygen to hydrogen and no other parts) then Putnam's argument wouldn't hold. (This is where p-zombies fail: it is stipulated that there is no experiment that can tell the difference.)
Putnam's main point was that we can be mistaken about what a thing is. Moreover, that when we can have two things (call them A and B) that we think are of the same type that we can not only be mistaken that A and B are of the same type, but that A could fit the type and B might not.
If this seems incredibly basic... it is. People make a big deal about it because prior to Putnam (and sometimes afterward) philosophers were saying crazy things like "the meanings in our heads don't have to refer to anything in the world," which essentially translates to "I can make a word mean anything I want!"
I agree with this to the extent that we shouldn't make the mistake that just because we have a model of something in our head means that our model corresponds to the real world. It's even stickier, because when a model doesn't conform we often keep the words around because they can be useful descriptions of the new thing we've fround. That can create confusion, especially during a period of transition. (Imagine someone saying that "Water cannot be H2O, because it is necessarily an Aristotelian element.") But thought experiments are very, very useful since all a "thought experiment" really is, is when you use the information already in your head and say, "Given what I already know, what do I think would happen in this circumstance?"
I think he's making a slightly different point. His point is that the reference of a term, which determines whether, say, the setence "Water is H2O" is true or not, depends on the environment in which that term came to be used. And this could be true even for speake... (read more)