Ian, there's nothing wrong with reductionism.

Overly simplistic reductionism is wrong, e.g., if you divide a computer into individual bits, each of which can be in one of two states, then you can't explain the operation of the computer in just the states of its bits. However, that reduction omitted an important part, the interconnections of the bits--how each affects the others. When you reduce a computer to individual bits and their immediate relationships with other bits, you can indeed explain the whole computer's operation, completely. (It just becomes unwieldy to do so.)

"I mean if you list all the actions that it's parts can do alone, the combined thing can have actions that aren't in that list."

What are these "actions that aren't in that list"? They are still aggregations of interactions that take place at a lower level, but we assign meaning to them. The extra "actions" are in our interpretations of the whole, not in the parts or the whole itself.

Eliezer, I guess the answer you want is that "science" as we know it has at least one bias: a bias to cling to pragmatic pre-existing explanations, even when they embody confused thinking and unnecessary complications. This bias appears to produce major inefficiencies in the process.

Viewing science as a search algorithm, it follows multiple alternate paths but it only prunes branches when the sheer bulk of experimental evidence clearly favours another branch, not when an alternate path provides a lower cost explanation for the same evidence. For efficiency, science should instead prune (or at least allocate resources) based on a fair comparison of current competing explanations.

Science has a nostalgic bias.

Suggested reading: http://en.wikipedia.org/wiki/Visual_cortex#Function "Conceptually, this retinotopy mapping is a transformation of the visual image from retina to V1. The correspondence between a given location in V1 and in the subjective visual field is very precise: even the blind spots are mapped into V1."

We can easily reject the Cartesian theater notions, but there are still "paintings" of images in our brains.

"This pattern of belief is very hard to justify from a Bayesian perspective. It is just the same hypothesis in both cases. Even if, in the second case, I announce an experimental method and my intent to actually test it, I have not yet experimented and I have not yet received any observational evidence in favor of the hypothesis."

Always consider the source.

It is the same hypothesis, but if the person telling the story is unknown to me at the outset, then the credibility of the source rises with test plans, as follows.

After the just-so story: This could be any crank making up stories to sell an idea.

After hearing of a possible test: If this person has taken the time to work out how they would test it, then my confidence is significantly increased. They've just set themselves apart from the pure story-tellers.

After looking at the thoroughness of their proposed tests: Confidence rises if they've shown a willingness to consider lots of ways that their story and testing could be wrong or misleading. Evidence of familiarity with and adherence to scientific methodology does increase credibility.

After hearing that they are committed to performing their tests: Confidence increases a little more. Not much, but a little.

Sure, these increases in confidence are small--tiny compared to the boost from actually performing even one of the tests--but they are not zero.

If I'm already familiar with the story-teller, and have some prior opinion of their experience and capability, then you're right: hearing about the tests shouldn't affect my confidence.