EDIT: I'm moving this to the Discussion section because people seem to not like it (lack of upvotes) and to find the writing unclear. I'd love writing advice, if anyone wants to offer some.
Related to: Dissolving the question, Explaining vs explaining away
I review parts of the reductionism sequence, in hopes of setting up for future reduction work.
So, you’ve been building up your reductionism muscles, on LW or elsewhere. You’re no longer confused about a magical essence of free will; you understand how particular arrangements of atoms can make choices. You’re no longer confused about a magical essence of personal identity; you understand where the feeling of “you” comes from, and how one could in principle create many copies of you that continued "your" experiences, and how the absence of an irreducible essence doesn’t reduce life’s meaning.
The natural next question is: what other phenomena can you reduce? What topics are we currently confused about which may yield to the same tools? And what tools, exactly, do we have for such reductions?
With the goal of paving the way for new reductions, then, let’s make a list of questions that persistently felt like questions about magical essences, including both questions that have been solved, and questions about which we are currently confused. And let’s also list tools or strategies that assisted in their dissolution. I made an attempt at such lists below; perhaps you can help me refine them?
Some places where many expected a fundamental or non-reducible “essence”[1]:
1. Ducks.
Why it's tempting to postulate an essence: Organisms seem to come in types. New organisms of the given type (e.g., new ducks) come into existence, almost as though the the type “Duck” had causal power. Humans are able to form mental concepts of "duck" that approximately mirror the outside predictive regularities.
2. Life.
Why it's tempting to postulate an essence: Living creatures act very differently from dead creatures. A recently killed animal doesn’t move, loses its body heat, etc., even though its matter is in almost the same configuration. [2]
3. Free will.
Why it's tempting to postulate an essence: Humans (among other things) are in fact organized to “choose” their actions in some meaningful sense. We (mostly) choose a single course of action in a relatively unified manner that responds to outside information and incentives. “Choice” also seems like a useful internal concept, but I’m not sure how to describe the details here.
4. Personal identity
Why it's tempting to postulate an essence: People have personalities, plans, beliefs, bodies, etc. that approximately persist over time. Internally, we experience consistent memories that happened “to us”, we choose our own actions, and we anticipate future experiences.
5. Pain
Why it's tempting to postulate an essence: We feel pain. We find ourselves motivated to avoid pain. We sometimes almost feel others’ pain, as when we wince and rub our thumbs after watching someone else smash their thumb with a hammer, and we often find ourselves motivated to avoid their pain as well. We can report verbally on the pain, modify our behavior to reduce the pain, etc.
6. Mathematics
Why it's tempting to postulate an essence: Mathematics often pops up in science. It’s also “simple”, is at least somewhat culturally universal, is relatively easy to implement portions of in machines, has a nice notion of “proof” whereby we can often formally determine what is true, and can often determine true results without much contact with outside empirical data, and is something aliens might plausibly share with us.
7. Reality / existence / the physical world
Why it's tempting to postulate an essence: Our perceptions are well predicted by imagining a set of fairly stable objects that we can see, touch, turn over in our hands, etc. and that retain their color, shape, heft, and other properties fairly stably over time. At higher levels of abstraction, too, the world is fairly lawful and coherent. [3]
What lessons for future reductions?
These examples suggest the following heuristics:
A. Even when it really, really feels like there should be an essence, there probably isn’t one.
B. Philosophical questions are just ordinary questions that one is particularly ignorant about; they are not questions about separate magisteria that must permanently be reasoned about in some special way.
C. People expect magical essences in places where there really are interesting empirical regularities. In order to understand those regularities, and to create a new set of concepts that better do the work that our old magical essences intuitions used to do, it is necessary to do real research. Ritual assertions that “It’s all physics” and “there aren’t essences” do not create the needed concepts and anticipations.
D. A reasonable first step, in tackling a new confusion, is to ask the why it feels like there is a question or concept there, and to list the empirical regularities, or cognitive artifacts, that contribute to that feeling.
These heuristics aren't original; Eliezer noted them already in his reductionism sequence (which is very much worth reading). But I suspect that many apply these heuristics more to problems they already understand (“of course free will has no magical essence”) more than to problems we don’t yet understand (“of course there is no magical essence that distinguishes our actual, real world from imaginable physicses/worlds that aren't real").
I'm hoping that reviewing heuristics for reduction, and staring at solved and unsolved problems side by side, may help us with the unsolved problems (which I'll attempt some steps toward in subsequent posts).
[1] I agree with SarahC’s point that humans seem predisposed to impute essences everywhere. Still, discussions about whether there’s a magical essence “free will” seem to pop up more often than discussions about whether there’s a magical essence “carpet”, “ocean”, “California”, or “female”. I mean, folks are interested in these other questions, and they have discussions about what meaning to use and how much that meaning cleaves nature at its joints, but they don’t generally expect a separate sort of essence that has causal powers and isn’t made out of atoms.
[2] People unacquainted with modern biology seem often to make predictive errors due to expecting an essence of life. For example, I had lunch the other day with a physics professor from a good university who thought that, even if we could assemble an atom-for-atom duplicate of a person's exact physical state, it might well not act like a person for want of a soul. Another acquaintance was surprised to hear that scientists do in fact believe that a cell assembled in a test tube would act just like a cell assembled anywhere else.
[3] I'm less satisfied with this unpacking than with the others on the list. Can anyone do better?
I have been vaguely thinking of writing a top level article about psychological essentialism for a while, but this seems like a good place to just point to it. There is a large body of research on the subject and the best summary of it I know is Susan Gelman's "Essentialism in Everyday Thought".
The process of imputing causation to visible characteristics from an unseen inner variable seems to be something humans subconsciously and automatically do in some situations. You can do studies of essentialist reasoning in children by asking "Suppose an X was raised by Ys, in case C would the X tend to do this X-ish things or this Y-ish things?" Very young children will give different answers for difference values of C, and in some cases (like with human language) they are clearly wrong which makes it seem like maybe they are executing a domain specific "complex essentialist cognitive module" rather than reasoning from observed evidence.
If you want to see some back and forth on the subject instead of just taking one author's word for it, here is a 1993 criticism of the essentialist research program by Susan Jones and Linda Smith and a rebuttal by Susan Gelman.
My impression is that essentialist cognition is enormously common because it is enormously useful. There is an essence for ducks and its called the duck genome. Hypothesized essences have been mapped to one or more physical mechanisms so often that mostly I just assume that this can be done for all essences that are coherent. The mechanisms generally have approximately the pragmatic properties normally ascribed to essences (like immutability) but it doesn't match human intuitions perfectly because, whether its a democracy or an ecology or a painting "it's just an arrangement of atoms, right?"
After I internalized this view of the world, I gained a measure of sympathy for people who believed in supernatural theories because those situations became intelligible to me as people having working models for reality using the built in tools of human reasoning. "Evil ghosts in the swamp" model "malaria" well enough to explain why to avoid swamps while looking for food. "God" models "inscrutable optimization processes stronger than me" enough to help people make a kind of peace with the fact of limited power and explain the appearance of order in the world while looking for food.
When I need to "argue past the essence" without giving the whole cogsci backstory I try to point out the "magic sparks" buried in the theory and then denaturalize the sparks by hypothesizing vaguely realistic complex mechanisms in their place. Like sometimes people deploy the concept of IQ as working like a magic spark that is more or less bright, and you can denaturalize it by breaking it into pieces that can obviously vary independently like working memory capacity, vocabulary acquisition, "good nutrition, good sleep, and the time to cultivate your mind", and so on.
But most of the time, a theory with a spark is perfectly good for making the predictions you need to avoid scary situations and find food. I would expect an AGI worth its salt to have some internal mechanism that functions roughly to "notice the sparks" and use them as appropriate and push deeper as dictated by opportunity and need :-)
Essentialism also seems very prominent in human pleasure, per "how pleasure works" book.
As far as supernatural theories, I am in an interesting position as far as taking some meditation training from a martial arts teacher; on one hand I (surely hope!) am not "aligning meridians of the body" while doing the breathing exercises, on the other hand I don't want to dismiss this incorrect "model" too early as it might be of further usefulness, given that these systems tend to be very ancient.