This argument is not going to win over their heads and hearts. It's clearly written for a reductionist reader, who accepts concepts such as Occam's Razor and knowing-what-a-correct-theory-looks-like.

I would suggest that people who have already studied this issue in depth would have other reasons for rejecting the above blog post. However, you are right that philosophers in general don't use Occam's Razor as a common tool and they don't seem to make assumptions about what a correct theory "looks like."

If conceivability does not imply logical possibility, then even if you can imagine a Zombie world, it does not mean that the Zombie world is logically possible.

Chalmers does not claim that p-zombies are logically possible, he claims that they are metaphysically possible. Chalmers already believes that certain atomic configurations necessarily imply consciousness, by dint of psychophysical laws.

The claim that certain atomic configurations just are consciousness is what the physicalist claims, but that is what is contested by knowledge arguments: we can't really conceive of a way for consciousness to be identical with physical states.

The "conceivability" of zombies is accepted by a substantial fraction, possibly a majority, of academic philosophers of consciousness.

This can be made precise. According to the 2009 PhilPapers Survey (sent to all faculty at the top 89 Ph.D-granting philosophy departments in the English-speaking world as ranked by the Philosophical Gourmet Report, plus 10 high-prestige non-Anglophone departments), about 2/3 of professional philosophers of mind think zombies are conceivable, though most of these think physicalism is true anyway. Specifically, 91 of the 191 respondents (47.6%) said zombies are conceivable but not metaphysically possible; 47 (24.6%) said they were inconceivable; 35 (18.3%) said they're (conceivable and) metaphysically possible; and the other 9.4% were agnostic/undecided or rejected all three options.

Looking at professional philosophers as a whole in the relevant departments, including non-philosophers-of-mind, 35.6% say zombies are conceivable, 16% say they're inconceivable, 23.3% say they're metaphysically possible, 17% say they're undecided or insufficiently familiar with the issue (or they skipped the question), and 8.2% rejected all three options. So the average top-tier Anglophone philosopher of mind is more likely to reject zombies than is the average top-tier Anglophone philosopher. (Relatedly, 22% of philosophers of mind accept or lean toward 'non-physicalism', vs. 27% of philosophers in general.)

There is a stuff of consciousness which is not yet understood, an extraordinary super-physical stuff that visibly affects our world; and this stuff is what makes us talk about consciousness.

Chalmers' core objection to interactionism, I think, is that any particular third-person story you can tell about the causal effects of consciousness could also be told without appealing to consciousness. E.g., if you think consciousness intervenes on the physical world by sometimes spontaneously causing wavefunctions to collapse (setting aside that Chalmers and most LWers reject collapse...), you could just as easily tell a story in which wavefunctions just spontaneously collapse without any mysterious redness getting involved; or a story in which they mysteriously collapse when mysterious greenness occurs rather than redness, or when an alien color occurs.

Chalmers thinks any argument for thinking that the mysterious redness of red is causally indispensable for dualist interactionism should also allow that the mysterious redness of red is an ordinary physical property that's indispensable for physical interactions. Quoting "Moving Forward on the Problem of Consciousness":

The real "epiphenomenalism" problem, I think, does not arise from the causal closure of the physical world. Rather, it arises from the causal closure of the world! Even on an interactionist picture, there will be some broader causally closed story that explains behavior, and such a story can always be told in a way that neither includes nor implies experience. Even on the interactionist picture, we can view minds as just further nodes in the causal network, like the physical nodes, and the fact that these nodes are experiential is inessential to the causal dynamics. The basic worry arises not because experience is logically independent of physics, but because it is logically independent of causal dynamics more generally.

The interactionist has a reasonable solution to this problem, I think. Presumably, the interactionist will respond that some nodes in the causal network are experiential through and through. Even though one can tell the causal story about psychons without mentioning experience, for example, psychons are intrinsically experiential all the same. Subtract experience, and there is nothing left of the psychon but an empty place-marker in a causal network, which is arguably to say there is nothing left at all. To have real causation, one needs something to do the causing; and here, what is doing the causing is experience.

I think this solution is perfectly reasonable; but once the problem is pointed out this way, it becomes clear that the same solution will work in a causally closed physical world. Just as the interactionist postulates that some nodes in the causal network are intrinsically experiential, the "epiphenomenalist" can do the same.

This brings up a terminology-ish point:

The technical term for the belief that consciousness is there, but has no effect on the physical world, is epiphenomenalism.

Chalmers denies that he's an epiphenomenalist. Rather he says (in "Panpsychism and Panprotopsychism"):

I think that substance dualism (in its epiphenomenalist and interactionist forms) and Russellian monism (in its panpsychist and panprotopsychist forms) are the two serious contenders in the metaphysics of consciousness, at least once one has given up on standard physicalism. (I divide my own credence fairly equally between them.)

Quoting "Moving Forward" again:

Here we can exploit an idea that was set out by Bertrand Russell (1926), and which has been developed in recent years by Grover Maxwell (1978) and Michael Lockwood (1989). This is the idea that physics characterizes its basic entities only extrinsically, in terms of their causes and effects, and leaves their intrinsic nature unspecified. For everything that physics tells us about a particle, for example, it might as well just be a bundle of causal dispositions; we know nothing of the entity that carries those dispositions. The same goes for fundamental properties, such as mass and charge: ultimately, these are complex dispositional properties (to have mass is to resist acceleration in a certain way, and so on). But whenever one has a causal disposition, one can ask about the categorical basis of that disposition: that is, what is the entity that is doing the causing?

One might try to resist this question by saying that the world contains only dispositions. But this leads to a very odd view of the world indeed, with a vast amount of causation and no entities for all this causation to relate! It seems to make the fundamental properties and particles into empty placeholders, in the same way as the psychon above, and thus seems to free the world of any substance at all. It is easy to overlook this problem in the way we think about physics from day to day, given all the rich details of the mathematical structure that physical theory provides; but as Stephen Hawking (1988) has noted, physical theory says nothing about what puts the "fire" into the equations and grounds the reality that these structures describe. The idea of a world of "pure structure" or of "pure causation" has a certain attraction, but it is not at all clear that it is coherent.

So we have two questions: (1) what are the intrinsic properties underlying physical reality?; and (2) where do the intrinsic properties of experience fit into the natural order? Russell's insight, developed by Maxwell and Lockwood, is that these two questions fit with each other remarkably well. Perhaps the intrinsic properties underlying physical dispositions are themselves experiential properties, or perhaps they are some sort of proto-experiential properties that together constitute conscious experience. This way, we locate experience inside the causal network that physics describes, rather than outside it as a dangler; and we locate it in a role that one might argue urgently needed to be filled. And importantly, we do this without violating the causal closure of the physical. The causal network itself has the same shape as ever; we have just colored in its nodes.

This ideas smacks of the grandest metaphysics, of course, and I do not know that it has to be true. But if the idea is true, it lets us hold on to irreducibility and causal closure and nevertheless deny epiphenomenalism. By placing experience inside the causal network, it now carries a causal role. Indeed, fundamental experiences or proto-experiences will be the basis of causation at the lowest levels, and high-level experiences such as ours will presumably inherit causal relevance from the (proto)-experiences from which they are constituted. So we will have a much more integrated picture of the place of consciousness in the natural order.

This is also (a more honest name for) the non-physicalist view that sometimes gets called "Strawsonian physicalism." But this view seems to be exactly as vulnerable to your criticisms as traditional epiphenomenalism, because the "causal role" in question doesn't seem to be a difference-making role -- it's maybe "causal" in some metaphysical sense, but it's not causal in a Bayesian or information-theoretic sense, a sense that would allow a brain to nonrandomly update in the direction of Strawsonian physicalism / Russellian monism by computing evidence.

I'm not sure what Chalmers would say to your argument in detail, though he's responded to the terminological point about epiphenomenalism. If he thinks Russellian monism is a good response, then either I'm misunderstanding how weird Russellian monism is (in particular, how well it can do interactionism-like things), or Chalmers is misunderstanding how general your argument is. The latter is suggested by the fact that Chalmers thinks your argument weighs against epiphenomenalism but not against Russellian monism in this old LessWrong comment.

It might be worth e-mailing him this updated "Zombies" post, with this comment highlighted so that we don't get into the weeds of debating whose definition of "epiphenomenalism" is better.

I removed the second post (What's in a Name?) from the list because it's been... well, debunked. From a recent SSC link post:

A long time ago I blogged about the name preference effect – ie that people are more positively disposed towards things that sound like their name – so I might like science more because Scott and science start with the same two letters. A bunch of very careful studies confirmed this effect even after apparently controlling for everything. Now Uri Simonsohn says – too bad, it’s all spurious. This really bothers me because I remember specifically combing over these studies and finding them believable at the time. Yet another reminder that things are worse than I thought.

One common question we hear about alignment research runs analogously to: "If you don't develop calculus, what bad thing happens to your rocket? Do you think the pilot will be struggling to make a course correction, and find that they simply can't add up the tiny vectors fast enough? That scenario just doesn't sound plausible."

Actually, that sounds entirely plausible.

The case is similar with, e.g., attempts to develop theories of logical uncertainty. The problem is not that we visualize a specific AI system encountering a catastrophic failure because it mishandles logical uncertainty; the problem is that all our existing tools for describing the behavior of rational agents assume that those agents are logically omniscient, making our best theories incommensurate with our best practical AI designs.

Well, of course, part of the problem is that the best theories of "rational agents" try to assume Homo Economicus into being, and insist on cutting off all the ways in which physically-realizable minds cannot fit. So we need a definition of rationality that makes sense in a world where agents don't have completed infinities of computational power and can be modified by the environment and don't come with built-in utility functions that necessarily map physically realizable situations to the real numbers.

If we could program that computer to reliably achieve some simple goal (such as producing as much diamond as possible), then a large share of the AI alignment research would be completed.

Wait wait wait. You're saying that the path between Clippy and a prospective completed FAI is shorter than the path between today's AI state-of-the-art and Clippy? Because it sounds like you're saying that, even though I really don't expect you to say that.

On the upside, I do think we can spell out a research program to get us there, which will be grounded in current computational cog-sci and ML literature, which will also help with Friendliness/alignment engineering, which will not engender arguments with Jessica over math this time.

But now for the mandatory remark: you are insane and will kill us all ;-), rabble rabble rabble.

Update: We've hit our first fundraising target! We're now nearing the $200k mark. Concretely, additional donations at this point will have several effects on MIRI's operations over the coming year:

• There are a half-dozen promising people we'd be hiring on a trial basis if we had the funds. Not everyone in this reference class can be sent to e.g. the Oxford and Cambridge groups working on AI risk (CSER, FHI, Leverhulme CFI), because those organizations have different hiring criteria and aren't primarily focused on our kind of research. This is one of the main ways a more successful fundraiser this month translates into additional AI alignment research on the margin.

This includes several research fellows we're considering hiring (with varying levels of confidence) in the next year. Being less funding-constrained would make us more confident that our growth is sustainable, causing us to move significantly faster on these hires.

• Secondarily, additional funds would allow us to run additional workshops and run longer and meatier fellows/scholars programs.

Larger donations would allow us to expand the research team size we're shooting for, and also spend a lot more time on academic outreach.

Update from Ruairi Donnelly, the Executive Director at Raising for Effective Giving: "Ferruell's matching drive has been going on for a while behind the scenes, as he had a preference for doing it in the Russian poker community first. No limit was declared (but the implicit understanding was 'in the tens of thousands'). As donations were slowing down, our ambassador Liv Boeree shared it on Twitter. Ferruell has now confirmed that he'll match donations up to $50,000. [Roughly] $30,000 have already come in."

Announcement: Raising for Effective Giving just let us know that for the next two days, donations to MIRI through REG's donation page (under "Pick a specific charity: MIRI") will be matched dollar-for-dollar by poker player Ferruell! Fantastic.

Announcement: Raising for Effective Giving just let us know that for the next two days, donations to MIRI through REG's donation page (under "Pick a specific charity: MIRI") will be matched dollar-for-dollar by poker player Ferruell! Fantastic.

• 1 - Humans can't reliably precommit. Even if they could, precommittment is different than using an "acausal" decision theory. You don't need precommitment to one-box in Newcomb's problem, and the ability to precommit doesn't guarantee by itself that you will one-box. In an adversarial game where the players can precommit and use a causal version of game theory, the one that can precommit first generally wins. E.g. Alice can precommit to ignore Bob's threats, but she has no incentive to do so if Bob already precommitted to ignore Alice's precommitments, and so on. If you allow for "acausal" reasoning, then even having a time advantage doesn't work: if Bob isn't born yet, but Alice predicts that she will be in an adversarial game with Bob and Bob will reason acausally and therefore he will have an incentive to threaten her and ignore her precommitments, then she has an incentive not to make such precommitment.
• 2 - This implies that the future AI uses a decision theory that two-boxes in Newcomb's problem, contradicting the premise that it one-boxes.
• 3 - This implies that the future AI will have a deontological rule that says "Don't blackmail" somehow hard-coded in it, contradicting the premise that it will be an utilitarian. Indeed, humans may want to build an AI with such constants, but in order to do so they will have to consider the possibility of blackmail and likely reject utilitarianism, which was the point of Roko's argument.
• 4 - Shut up and multiply.

Assuming you for some reason are following a decision theory that does put you at risk of acausal blackmail: Since the hypothetical agent is superintelligent, it has lots of ways to trick people into thinking it's going to torture people without actually torturing them. Since this is cheaper, it would rather do that. And since we're aware of this, we know any threat of blackmail would be empty.

Um, your conclusion "since we're aware of this, we know any threat of blackmail would be empty" contradicts your premise that the AI by virtue of being super-intelligent is capable of fooling people into thinking it'll torture them.