If you are a superscientist, there is nothing you can learn from running a programme that you cannot get from examining the code.
This seems trivially false. See also the incomputability of pure Solomonoff induction.
Likewise, I see no reason to expect that a mathematical process could look at a symbolic description of itself and recognize it with intuitive certainty. We have some reason to think the opposite. So why expect to recognize "qualia" from their descriptions?
As orthonormal points out at length, we know that humans have unconscious processing of the sort you might expect from this line of reasoning. We can explain how this would likely give rise to confusion about Mary's Room.
So why expect to recognize "qualia" from their descriptions?
Why expect an inability to figure out some things about your internal stare to put on a techinicolor display? Blind spots don't look like anything. Not even perceivable gaps in the visual field.
[Cross-posted.]
1. Defining the problem: The inverted spectrum
A. Attempted solutions to the inverted spectrum.
B. The “substitution bias” of solving the “easy problem of consciousness” instead of the “hard problem.”
2. The false intuition of direct awareness
A. Our sense that the existence of raw experience is self-evident doesn’t show that it is true.
B. Experience can’t reveal the error in the intuition that raw experience exists.
C. We can’t capture the ineffable core of raw experience with language because there’s really nothing there.
D. We believe raw experience exists without detecting it.
3. The conceptual economy of qualia nihilism pays off in philosophical progress
4. Relying on the brute force of an intuition is rationally specious.
Against these considerations, the only argument for retaining raw experience in our ontology is the sheer strength of everyone’s belief in its existence. How much weight should we attach to a strong belief whose validity we can't check? None. Beliefs ordinarily earn a presumption of truth from the absence of empirical challenge, but when empirical challenge is impossible in principle, the belief deserves no confidence.