Eliezer_Yudkowsky said:

It is only the Mind Projection Fallacy that makes some people talk as if the higher levels could have a separate existence - different levels of organization can have separate representations in human maps, but the territory itself is a single unified low-level mathematical object. Suppose this were wrong. Suppose that the Mind Projection Fallacy was not a fallacy, but simply true. Suppose that a 747 had a fundamental physical existence apart from the quarks making up the 747. What experimental observations would you expect to make, if you found yourself in such a universe? If you can't come up with a good answer to that, it's not observation that's ruling out "non-reductionist" beliefs, but a priori logical incoherence. If you can't say what predictions the "non-reductionist" model makes, how can you say that experimental evidence rules it out?

This comes from a post from almost a year ago, Excluding the Supernatural. I quote it because I was hoping to revive some discussion on it: to me, this argument seems dead wrong.

The counter-argument might go like this:

Reductionism is anything but a priori logically necessary-- it's something that must be verified with extensive empirical data and inductive, probabilistic reasoning. That is, we observe that the attributes of many entities can be explained with laws describing their internal relations. Occam's razor tells us that we don't need both the higher and lower order model to actually exist, so we unify our theory. The repeated experience of this success leads us to extrapolate that this can be done with all entities. Perhaps some entities present obstacles to this goal, but we then infer that their irreducibility is in the map (our model for understanding them) not in the territory (the entity itself.) But again, we infer this by assuring ourselves that they just haven't been explained YET--which implies it's reasonable, based on inductive reasoning from the past, to assume that they will be reduced. Or we describe some element of the entity's complexity that makes "irreducibility in practice" something to be expected. We therefore preserve its reducibility in principle.

But we do not (it seems to me) merely exclude its irreducibility based on a priori necessity. Why would we? It's perfectly conceivable. Eliezer describes in an earlier post the "small, hard, opaque black ball" that is a non-reductionist explanation of an entity. He claims its just a placeholder, something that fools us into thinking there's a causal chain where nothing has actually been clarified.

But it's perfectly conceivable that such a "black ball" could exist. I suppose there's no way to prove that it's irreducible, and not just unreduced as of yet, in the same way that one can't prove a negative. But this just presupposes that the default position ought to be reductionism. We should assume innocent until proven guilty. But which is innocent in this case: reducible or non-reducible?

So what if we come across something that appears to be a "black ball"? We attempt with all our mental and technological acuity to analyze it in terms or more fundamental laws, and every attempt fails. I would argue this is a good example of empirical evidence against materialist reductionism. We indeed have an entity that obeys laws which we can describe and predict--it just has laws that can't be reconciled with the physical laws of everything else, and when interacting with anything else, violates them.

Occam's razor is indeed strong here: we recognize that, given the faintest hope of reduction, we should throw out irreducibility in favor of having as few types of "stuff" as possible. This happens in the case of "elan vital." But it seems perfectly conceivable to me that there might be an entity that's truly a black ball.

But now we get to the dilemma: if the staid conventional normal boring understanding of physics and the brain is correct, there's no way in principle that a human being can concretely envision, and derive testable experimental predictions about, an alternate universe in which things are irreducibly mental. Because, if the boring old normal model is correct, your brain is made of quarks, and so your brain will only be able to envision and concretely predict things that can predicted by quarks. You will only ever be able to construct models made of interacting simple things. People who live in reductionist universes cannot concretely envision non-reductionist universes. They can pronounce the syllables "non-reductionist" but they can't imagine it.

Now this seems so massively incorrect that I fear I'm misunderstanding Eliezer. Does anyone have any feedback? I'd love to make a post about this, once I generate some karma.