IAWYC, but let me point out that you are describing your position rather than supporting it.
As a way of supporting it, imagine that there's some other universe with physics that encapsulate ours: the Dark Lords of the Matrix can cheaply run a faithful simulation of anything that happens in this universe on that universe's computers. It's clear to me that, given the setup you two are discussing, the Dark Lords would see that those extra bits aren't doing anything at all; they can be removed without altering anything that they could observe in the simulation.
Now, anything we can observe about our conscious experience is assumed to have an effect on our brains as we think of it, and thus the Dark Lords could observe it as well. (Namely, if there's a distinction between a "physically real" world and another one that's just "mathematically possible" with an identical copy of you, there's no difference from the Dark Lords' perspective between the description of your brain thinking "But I really exist!" and the description of your copy's brain thinking the same thing.) Note that this is an instance of the GAZP in action.
So by Occam's Razor, I don't think there's a justification for adding extra bits to the way the universe is described, when literally nobody within or outside the universe can be pointed to as having justification that those bits are one way or another.
It has been well over a year since I first read Permutation City and relating writings on the internet on Greg Egan's dust theory. It still haunts me. The theory has been discussed tangentially in this community, but I haven't found an article that directly addresses the rationality of Egan's own dismissal of the theory.
In the FAQ, Egan says things like:
and:
Isn't this, along with so many other problems, a candidate for our sometime friend the anthropic principle? That is: only in a conscious configuration field which has memories of perceptions of an orderly universe is the dust theory controversial or doubted? In the vastly more numerous conscious configuration fields with memories of perceptions of a chaotic and disorderly universe lacking a rational way to support the observer the dust theory could be accepted a priori or at least be a favored theory.
It is fine to dismiss dust theory because it simply isn't very helpful and because it has no predictions, testable or otherwise. I suppose it is also fine never to question the nature of consciousness as the answers don't seem to lead anywhere helpful either; though the question of it will continue to vex some instances of these configuration states.