Thanks for the explanation.
The idea is that when you are listening to music, you are handicapping yourself by taking some of the attention of the aural modality.
I'd heard something similar from a friend who majored in psychology, but they explained it in terms of verbal processing rather than auditory processing more generally, which is why (they said) music without words wasn't as bad.
I'm not sure whether it's related, but I've also been told by a number of musically-trained friends that they can't work with music at all, because they can't help but analyse it as they listen: for them, listening seems to automatically involve processing work that it doesn't (seem to) for me, precisely because I'm not capable of such processing. (This was part of the reason I was originally wondering about individual variation; the point you make at the end is really interesting in this regard too.)
In a possibly-related anecdote, I can't listen to music I've played in Guitar Hero while working, as my mind switches into Guitar Hero mode and all I see are streams of colored buttons.
While working with Marcello on AI this summer, I've noticed that I have some standard mantras that I invoke in my inner dialogue (though only on appropriate occasions, not as a literal repeated mantra). This says something about which tropes are most often useful - in my own working life, anyway!
(When encountering a need to discard some idea to which I was attached, or admit loss of a sunk cost on an avenue that doesn't seem to be working out.)
(If the first mantra doesn't work; then I actually visualize the universe already being a certain way, so that I can see the penalty for being a universe that works a certain way and yet believing otherwise.)
(If getting too caught up in proposing solutions, or discouraged when solutions don't work out - the immediate task at hand is just to understand the problem, and one may ask whether progress has been made on this. From full understanding a solution usually follows quickly.)
(Try to get your mind involved and processing the various aspects of it.)
(If my mind seems to be going empty.)
(Focusing during work, or trying to load the problem into memory before going to sleep each night, in hopes of putting the subconscious to work on it.)
(My general visualization of the FAI problem; a huge, blank, impossibly high wall, which I have to run up as quickly as possible. Used to accomodate the sense of the problem being much larger than whatever it is I'm working on right now.)
(I first enunciated this as an explicit general principle when explaining to Marcello why e.g. one doesn't worry about people who have failed to solve a problem previously. When you actually solve the problem, those thoughts will predictably not have contributed anything in retrospect. So if your goal is to solve the problem, you should focus on the object-level problem, instead of worrying about whether you have sufficient status to solve it. The same rule applies to many other habitual worries, or reasoning effort expended to reassure against them, that would predictably appear as wasted motion in retrospect, after actually solving the problem.)
(A quote from C. J. Cherryh's Paladin, used when feeling rushed. I don't think it's true literally or otherwise, but it seems to convey an important wordless sentiment.
(When expecting the answer to go a particular way, or hoping for the answer to go a particular way, is exerting detectable pressure on an ongoing inquiry.)
I don't listen to music while working, because of studies showing that, e.g., programmers listening to music are equally competent at implementing a given algorithm, but much less likely to notice that the algorithm's output is always equal to its input. However, I sometimes think of the theme Emiya #0 when feeling fatigued or trying to make a special demand on my mind.