The issue is broached by Chalmers himself in The Conscious Mind (p. 201). He says:

... it is sometimes said that reference to an entity requires a causal connection to that entity; this is known as the causal theory of reference. If so, then it would be impossible to refer to causally irrelevant experiences.

He goes on to reject the causal theory of reference.

Here is a relevant excerpt from the SEP article on zombies:

But, arguably, it is a priori true that phenomenal consciousness, whether actual or possible, involves being able to refer to and know about one's qualia. If that is right, any zombie-friendly account faces a problem. According to the widely accepted causal theory of reference — accepted by many philosophers — reference and knowledge require us to be causally affected by what is known or referred to (Kripke 1972/80); and it seems reasonable to suppose that this too is true a priori if true at all. On that basis, in those epiphenomenalistic worlds whose conceivability seems to follow from the conceivability of zombies — (worlds where qualia are inert) — our counterparts cannot know about or refer to their qualia. That contradicts the assumption that phenomenal consciousness involves being able to refer to qualia, from which it follows that such epiphenomenalistic worlds are not possible after all. Hence zombies are not conceivable in the relevant sense either, since their conceivability leads a priori to a contradiction. To summarize: if zombies are conceivable, so are epiphenomenalistic worlds. But by the causal theory of reference, epiphenomenalistic worlds are not conceivable; therefore zombies are not conceivable.