I agree this is a factor in problem-solving. I've found it to be important too. However, I suspect this can't be the main reason behind the "magic genie" phenomenon because if it were, you'd expect that mind-quieting meditations between bursts of mathematical effort would be vastly more productive than spending hours on problems and taking breaks. E.g., spending fifteen minutes on a tough problem and then spending five minutes meditating, cycled three times, would produce vastly better results than thinking about the problem for an hour. I'm not aware that this is the case, nor that the 20% most prodigious mathematicians are above average in their interest in mindfulness.
I agree that it's not the main thing, but not with your analysis. For one, this "mindfulness" thing is never really unpacked well. It could be that the habit of not focusing and bouncing between ideas (that's considered "unmindful", right?) is what it takes to not get stuck in ruts, and that the helpful mindfulness related bit is meta-awareness that "I'm noticing that I'm stuck" - and then fixing it instead of freaking out about it.
Instead of practicing mindfulness by itself, I'd hold mindfulness as an ideal and attack the specific blocks more directly.
Oddly, it turns out that most mathematicians do not relate to the supposed experience of a solution presenting itself in a dream. Many, including Hadamard, have mentioned waking up to have the answer to a problem they've been working on trumpeted into mind, but not clearly as a result of dreamed experiences
This actually doesn't change my hypothesis much. I'm hypothesizing something that happens without requiring awareness. I occasionally notice myself making strange metaphorical connections that were there outside my awareness for some time and finally got bumped in. I very much expect this to happen completely outside awareness a lot. Heck, Milton Erickson was famous for doing this on purpose as a technique in therapy!
By and large, no. But in this domain, yes! [...]what I was talking about seemed to be the division between the sympathetic and parasympathetic nervous systems.
Which domain exactly?
Interesting. I'll have to look into that distinction more, and generally spend more time in that perspective. I have gotten similar benefits, just on a fairly small scale - it'd just allow me to make sense of things there were a bit elusive and import it back to the individual thoughts/feelings level model.
I'm currently testing a promising direction for a possible collection of units at CFAR. (For those who have attended some CFAR events or test sessions, this is a collection of refinements to the fudoshin/"panic" unit.) I've hit on what I think is a key puzzle whose answer might unlock a lot of the emerging art of rationality. I - and possibly most people here, eventually - would very much appreciate any insight you have to share.
The puzzle is how thought incubation works, ideally expressed in terms of neural systems or neuroanatomical structures. I'll first explain the phenomenon and then suggest the general reference class from which I'm hoping to get an answer.
The Phenomenon:
Mathematicians frequently report that often one of the most helpful things they can do to solve a problem they're stuck on is step away from it. Jacques Hadamard (1949) examined his own experiences and also talked to many of his colleagues to work out what the common structure of this experience was, and determined that there seems to be a fairly predictable sequence to it:
(1) Intensely focus on the problem, working through every permutation you can think of that's likely to produce an answer.
(2) Walk away from the problem and think about something else.
(3) The magic genie in your head might eventually, and often unexpectedly, yell a possible insight into your awareness.
For instance, Henri Poincaré reported struggling to work on Fuchsian functions over the course of several weeks and then being forced to walk away from the proof he had been stuck on due to a planned vacation. One day he was stepping onto a bus with his mind certainly not on mathematics, and suddenly the key insight he needed to finish the proof appeared in his mind. It was as though a part of his mind had been secretly working on the problem and then brought the finished product into his awareness. In this particular case it also came with a feeling of total confidence that verification would pan out (although Hadamard notes that the validation step after the insight is still essential because sometimes that feeling of total confidence is mistaken).
I definitely relate to this from when I was working on graduate mathematics. However, it also pattern-matches with other mental phenomena that are much more common. For instance, sometimes I think I know what a person's name is, but struggle as I might I can't quite remember it - and then a few minutes later after I've given up remembering the name the answer loudly announces itself, often quite out-of-context. Or when I'm trying to figure out a way of improving a throw in martial arts and then find the answer suddenly dawning on me at a random time.
I'm under the impression that this is a fairly universal kind of experience. I suspect you can think of examples in your own life where this has happened. ("Oh, now I remember where I put those keys!")
Reference Class for an Explanation:
I'm going to offer some overly simplistic examples of the kind of explanation I'm looking for. In this case, I think overly simplistic might be okay since I'm just trying to get a reasonable handle on how to munchkin the interaction between a few different neural systems. If it turns out that a more detailed and technically correct version is important, I'll probably dig into it (pending the VOI versus cost-of-information comparison).
There seems to be some evidence that one of the reasons children are as impulsive as they are is that they haven't yet developed their prefrontal cortices (PFCs) to the degree adults have. The prefrontal cortex seems to do at least two things: (1) hold long-term goals in mind and (2) engage executive function (i.e., halt orders on impulses, typically ones that don't match up with the long-term goals). This neuroanatomical structure seems to continue growing until sometime in one's early 20s - which might be why we also find that teenagers typically have less impulse control than twentysomethings but more than middle-schoolers, whereas we don't find such a clear distinction between twentysomethings and thirtysomethings. (Yes, this could also or even instead be cultural. I know it's complicated.) Incidentally, I understand that the PFC is also one of the neural structures most deactivated by alcohol - although my impression is that it shuts down the long-term goals thing and not the executive function. (This is based on my and others' experience that precommitment works perfectly well. It seems to me that saying things like "I couldn't help myself because I was drunk!" is more a social excuse than an actual explanation. But I'm only around 65% confident of this as a general claim.)
On a related note, it would seem that there's something in the same rough space as theory of mind that goes beyond the ability to pass the false-belief test. According to Rebecca Saxe, the capacity for empathy seems to come from a particular bit of the brain that doesn't finish growing until the mid-20s. Saxe also provides some evidence that a sufficiently strong and precisely directed magnet can basically deactivate that part of one's theory of mind temporarily. It seems quite plausible to me (though I really don't know) that activation of the sympathetic nervous system (SNS), such as in fight-or-flight reactions, decreases activation of this empathy part of the brain. This might be why, in a perceived crisis, some people switch to an almost tool-like view of others (e.g., knowing that overcoming the bystander effect requires pointing at a specific person and saying "You! Call 911!" but not really getting a sense in that moment of what that person's experience is like to be so singled out).
I'm quite aware that much of the above is speculation. I think speculation is fine, but having it grounded in some actual known neuroscience is ideal. That would give me something to dig into. E.g., if there's some reason to believe that this phenomenon is related to the enteric nervous system, I can start digging into the literature on that system to better understand how to munchkin its interactions with the (rest of the) autonomic nervous system.
An example of something outside the reference class I'm looking for is a "little man in the subconscious" explanation. I first read about this about twenty years ago as a model for how mental incubation works: you concentrate on a problem in order to communicate to a little man in your subconscious what you want to have done, and then you stop talking to him so he can go do what you just told him to do. Then he comes back with an answer once he's done, without regard to what you're doing when he's done. I agree that this seems to be a reasonable metaphor for what's going on, but it doesn't tell me for instance why the "little man" seems to respond so much more to SNS activity than parasympathetic activity, or why he can't go do his job once he has the instructions even if we continue to think about the problem.
More generally, psychodynamic "explanations" are unlikely to be helpful here. Talking about this as the "domain of the iNtuiting function" in reference to Jungian psychodynamic theory or Myers-Briggs won't tell me hardly anything about how this relates to stress oscillation.
So... Any suggestions about what this mysterious "little man" might actually be made of?