Does this phenomenon happen at all, or is it a product of selective memory? It seems to me that incidents where a solution occurs to us while we are thinking of something else will be more memorable (and make better stories) than incidents where we find a solution while explicitly working on it.
How would we distinguish a world in which stepping away from a problem makes a solution more probable to occur from a world in which stepping away from a problem makes a solution more awesome-feeling and therefore more likely to be remembered and repeated as a story?
You make a good point. Thank you for bringing it up!
The phenomenon definitely happens. There's no question that insights pop into mind out of context for at least some rather large chunk of the population. Ask just about any math professor or graduate student: I'm willing to give 85% odds that they'll indicate that key insights to problems they had been working on have occurred to them more than once during times when they weren't thinking about the problems in question.
Subjectively and in memory, I think there is a similarity between what happens when an insight pops into mind unexpectedly & out of context as compared to when it pops into mind in the middle of working on a problem. ("Oh! Right! I can just think of these geometric structures as elements of a basis for a vector field!") This is different in character from working out different permutations of the problem and coming across one that happens to work. It's possible that the method of derivation is basically the same but one is done automatically in the background - but I really don't know! I wouldn't be surprised one way or the other.
However, many, many mathematicians have stumbled on roughly the same strategy for dealing with tough problems: walk away from it and work on something else. There are lots and lots of surveys of mathematicians' reports of their working behavior, most of which jive with what I recall seeing while in graduate school. For instance, it's pretty typical for working mathematicians to have several largely unrelated problems they're tackling in parallel, and many of them say that this is specifically so that they can cycle through the problems and come back to them with fresh eyes, possibly with insight appearing in the pattern Hadamard describes.
This strikes me as Bayesian evidence for us being in the universe where stepping away increases the probability of solution over one where the narrative just happens to be interesting and compelling. Otherwise wouldn't mathematicians who just stick with one problem until it's done have better productivity overall?
Also, I want to be clear that I'm not claiming that all insight happens this way. I'm claiming that some insight does, and that it seems to play a pretty key role in a lot of human functioning and problem-solving. I don't need this method in order to solve a quadratic equation by hand, for instance, but I almost certainly would if I had to figure out (rather than look up) how to solve an arbitrary cubic equation.
The phenomenon definitely happens. There's no question that insights pop into mind out of context for at least some rather large chunk of the population. Ask just about any math professor or graduate student: I'm willing to give 85% odds that they'll indicate that key insights to problems they had been working on have occurred to them more than once during times when they weren't thinking about the problems in question.
The question was whether the insight is more likely to pop into mind when stepping away from the problem.
I seem to recall an experiment in which subjects told to think about a hard problem for two hours had a significantly lower success rate than subjects told to think about it for half an hour, go play sudoku for an hour, then think about it for another half an hour. But grep and google are coming up empty for me at the moment, so take this with a grain of salt.
Do you think that the point about mathematicians coming consistently to this strategy does not constitute evidence? It certainly isn't overwhelming evidence, but it seems suspicious that virtually all mathematicians who talk about mathematical process talk about the importance of walking away from problems. I'm personally not aware of a single mathematician who thinks that such a practice is unnecessary.
(My dissertation was on mathematicians' methods for navigating struggle in their research, and as part of that I did a fair amount of looking both at mathematicians' accounts and at summaries of such accounts. The closest thing to denying this phenomenon I've encountered is the strong insistance of a very tiny minority of mathematicians that "intuition" has nothing to do with mathematics - but those same people still reported needing multiple problems to work on in parallel so that they could turn their attention away from a given problem they were stuck on.)
Mathematicians' claims may too be explained by selective memory effects mentioned by fubarobfusco in the first comment in this thread. The question is how to discriminate between the case when the mathematicians' testimonies are reflecting an existing phenomenon and the case when they result from a bias. Even if the insights were less likely to materialise after stepping away, there would be plenty of cases of this happening, so the fact that virtually every mathematician can remember few of them wouldn't be surprising.
Even if the insights were less likely to materialise after stepping away, there would be plenty of cases of this happening, so the fact that virtually every mathematician can remember few of them wouldn't be surprising.
Point taken. I guess the likelihood ratio for this strategy being actively helpful is closer to 1 than I had previously thought.
However, it's not just a few incidences. It's remarkably frequent. And it's also still valuable to note that problem-solving can occur in the background without the need for conscious attention. Even if the background process turns out not to be as efficient as conscious reflection, freeing up attention while still working on the problem looks like an obvious win to me.
One thing that I'm pretty sure is going on and that might be a sufficient explanation is that it takes time to develop fluency in a hard problem. You can solve a simple problem in one go if you can hold enough of it in your mind to see the next steps of a plan that prove useful, and the same happens with the results of those steps, and eventually you reach a solution. For a harder problem, you might fail to see a specific plan, so you develop various observations about the problem and additional representations of its aspects, without having a clear sense of which of them will be useful, and there are too many of these to hold in your mind at the same time, as even though the new observations may be obtained by the methods you know well, they are in themselves new facts that are not yet thoroughly familiar.
To get to the next step, it might be necessary to be able to access a lot of these observations easily, without spending attention on recreating them. It takes time to familiarize yourself with the new observations (and with the way they connect to the original problem and to each other), to commit all these details to long term memory and to train your imagination to easily retrace the connections between them. But once you've done so, you obtain new superpowers with respect to that problem (and perhaps others analogous to it). A proof that would've taken 15 steps in terms of the ideas you had when you started working on the problem (and so wasn't apparent), now takes only 4 steps in terms of the new auxiliary ideas you've developed in the meantime, and you can see it at a glance. (Perhaps if your mind wanders when you're stepping onto a bus, and spends a few seconds on the problem, this proves sufficient to take advantage of the prior training and notice the solution.)
This analysis suggests that if you are stuck on a problem you need to solve, and you have enough time on your hands, (1) you should deliberately and systematically study the observations associated with your problem, even the ones that don't seem immediately useful, and those that are easy to obtain, until these observations and their connections to the rest become obvious without the need to concentrate on reconstructing them, (2) revisit all (relevant) parts of the problem when you expect that new observations have been internalized since the last time you've revisited the problem.
This seems very insightful to me. In physics, it's definitely my experience that over time I gain fluency with more and more powerful concepts that let me derive new things in much faster and simpler ways. And I find myself consciously working ideas over in my mind with, I think, the explicit goal of advancing this process.
The funny thing about this is that before I gain these "superpowers," I'll read an explanation in a textbook, which is in terms of high-level ideas that I haven't completely grasped yet, so the reading doesn't help as much as it should. The book claims, "this follows immediately from Lorentz invariance," and I don't really see what's going on. Then, later, after I've understood those ideas, I find myself explaining things to myself in much the same words as the textbook: "I see! It's simple! It follows immediately from Lorentz invariance!"--but now this really is an explanation, and the words have a lot more meaning.
I'm reminded of the Interdict of Merlin in HMPOR.
There's been a significant amount of work on this problem, if you are wanting what I think you are wanting. Quick googling got me:
Jogging, doing the dishes and riding the bus to work are other examples.
The results of neuroimaging studies indicate that these types of situations allow the outermost regions of the prefrontal cortex - those areas of the brain that help exert cognitive control - to loosen the reins and allow thought processes and neural activity not strictly related to the primary task.
In this state of defocused attention, one idea can trigger the next across a relatively unconstrained range of concepts and associations that might otherwise be viewed as completely unrelated. The resulting novel connections may explain how a broad focus can significantly facilitate creative thought and inventive problem-solving.
It may also explain how wandering thoughts about a thistle burr, a hymn-book marker and a falling apple could have inspired developments as useful as Velcro, Post-it notes and a theory of gravity.
And a wikipedia article on "incubation" ( http://en.wikipedia.org/wiki/Incubation_(psychology) )
Incubation is one of the 4 proposed stages of creativity: preparation, incubation, illumination, and verification.[1] Incubation is defined as a process of unconscious recombination of thought elements that were stimulated through conscious work at one point in time, resulting in novel ideas at some later point in time.[2]
The experience of leaving a problem for a period of time, then finding the difficulty evaporates on returning to the problem, or even more striking, that the solution "comes out of the blue", when thinking about something else, is widespread. Many guides to effective thinking and problem solving advise the reader to set problems aside for a time.
...Recent advances in neuroscience provide intriguing evidence of the mechanisms underlying incubation effects, particularly those that occur during sleep.
Warning: Annealing sounds impressive if you're new to machine learning but AFAIK it's pretty weak as machine-learning methods go, and the real algorithms at work in human creativity are going to involve algorithms more powerful than that - albeit something on the level of 'cognitive temperature' might still be playing a partial role somewhere.
That definitely seems to be what's going on! Any clue what the neural structures are that take the dominant role in this process?
No idea - it's just the analogy I use for myself :) I think of it like a continuation of the learning process, but I don't know if the brain actually represents solutions as a favored arrangement of cells, and I don't know if there's some changing noise source that would lead to something like annealing.
My suspicion about the mechanism is that as you work on a problem, you tend to get stuck in one particular approach. When you step away from the problem, you allow yourself the possibility of approaching it from a new angle. Being so focused on idea X before, could have prevented you from considering alternative ideas Y and Z. So suppose you're thinking about something else, but your brain then briefly flicks back to the problem. This time, ideas Y and Z popup, and it's suddenly obvious that if you combine them, that you'll get the solution. So, it isn't clear at all that you would need any kind of background processing to occur in order to observe this effect. It wouldn't surprise me at all if sleep helped here as we know it rearranges memories, but I've seen this effect without any sleep at all and I don't know enough neuroscience to know if there is any brain optimising going on when you're awake.
For purposes of an informed discussion, vide Daniel Schacter's The Seven Sins of Memory.
Note: The paper is old, but provides good background information for those who lack it. I can't think of an answer yet, but wanted to help others who may not know where to start. The file will deactivate if for thirty days none download the file.
To determine the exact rules of a system and exploit them in uncommon ways - in particular if the methods violate some sort of norms.
Thinking about it for a minute, I think the difference between "hacking” and “munchkin-ing”, insofar as there is one, is that the former is about the kind of results (unusual/unobvious) and the latter is about the scale of results (large relative to effort).
Yeah, I kind of missed it. Hack is sort of something you do to something. Munchkinning is more... ontological. You are the munchkin if your existence is oriented to this sort of thing.
But I suppose if you use it as a verb, it could basically be a more transgressive version of hack.
I have a couple speculations here, but the're not on the level you're asking for.
The first is that they're getting stuck because the dead end thoughts are becoming too salient and crowding out other thoughts. This certainly happens when getting stuck with emotional issues, and I strongly suspect the same thing is going on with math - especially if you're feeling frustrated and stubborn about it. This suggests the solution of reducing the saliency of those thoughts, and indeed I have had success with that method in physics tutoring (though that was an obviously distressing meta level thought)
The second is a bit more speculative. I think that the problem really is on your mind, just not at the forefront. Sorta like how your dreams reflect stuff you've been thinking about a lot even if you weren't actively thinking about it when you went to sleep. The way I think it works is that you're running into all sorts of patterns and you just happen to come across some with some isomorphism in it. The classic example of this is the benzene ring and the daydream about snakes eating their own tails. This suggests you start looking at things with varied structure that might be similar enough and not trying to make the connections. I haven't experimented with this. Low doses of psychedelics also up your pattern matching and make these connections more likely. I have experiemented with this, and while it has given me insights on problems I was working on and problems I wasn't working on, I don't remember any with a good control where I was totally stuck until trying a low dose psychedelic.
As far as the "looking for answers in neuroanatomical structures" bit, that's how I started my research into hypnosis, but I ended up giving up on that path. The research I could find was strong evidence that it was legit, but it wasn't that helpful for a couple reasons. For one, "structure X is involved in Y" leaves a whole lot unsaid as to how it does Y and gives little more than a general direction to throw ideas and see what sticks. And perhaps more importantly, my interventions aren't on the level of neuroanatomical structures.
I've gotten a whole lot more mileage out of looking at how thoughts and feelings relate to other thoughts and feelings, and not so much where they physically live. Have you actually found otherwise?
The first is that they're getting stuck because the dead end thoughts are becoming too salient and crowding out other thoughts.
I agree this is a factor in problem-solving. I've found it to be important too. However, I suspect this can't be the main reason behind the "magic genie" phenomenon because if it were, you'd expect that mind-quieting meditations between bursts of mathematical effort would be vastly more productive than spending hours on problems and taking breaks. E.g., spending fifteen minutes on a tough problem and then spending five minutes meditating, cycled three times, would produce vastly better results than thinking about the problem for an hour. I'm not aware that this is the case, nor that the 20% most prodigious mathematicians are above average in their interest in mindfulness.
I think that the problem really is on your mind, just not at the forefront.
I think you must be right! Something is processing the problem and producing a solution later down the road.
The classic example of this is the benzene ring and the daydream about snakes eating their own tails.
Yeah, I love that example! Oddly, it turns out that most mathematicians do not relate to the supposed experience of a solution presenting itself in a dream. Many, including Hadamard, have mentioned waking up to have the answer to a problem they've been working on trumpeted into mind, but not clearly as a result of dreamed experiences. Because of this (and also because of people like Stephen LaBerge talking about problem-solving in lucid dreams), I started digging into known examples of insights derived from dreams. This one was the only one I could find - and it turns out that it might not have ever happened!
I've gotten a whole lot more mileage out of looking at how thoughts and feelings relate to other thoughts and feelings, and not so much where they physically live. Have you actually found otherwise?
By and large, no. But in this domain, yes!
The unit that spawned this line of inquiry was originally based entirely on my experience in applying mindfulness to martial arts. In aikido, you can do some really remarkable things, some of which seem downright implausible, by slipping into an appropriately mindful state. This usually gets expressed very mystically ("You must have no sense of yourself, no desire to throw your opponent, and only then will they fall - but you must not strive for mindfulness with this as your goal"), so I figured I'd give a shot at operationalizing it when teaching aikido. This was really quite successful, allowing me to give people progress comparable (claims my brain from past dojo experience) to the first year of training in just a few months of weekly sessions.
This gave me some pretty good epistemic tools that have proven to be critical for me as I've grown as a rationalist. So, I thought to share them, and one of our wonderful volunteers (Dan Keys) pointed out to me that much of what I was talking about seemed to be the division between the sympathetic and parasympathetic nervous systems. As I dug into that, I found that the correspondence was surprisingly exact. Basically, it sounds like the Pareto efficiency of mindfulness consists of being in parasympathetic mode most of the time.
This allowed for some hacks that I found immediately useful. For instance, I knew that people expressed stress physically by doing things like rubbing their necks and breathing more shallowly, but now I know that the whole host of how the two branches of the autonomic nervous system oppose one another can offer signals for where one is on the PNS/SNS spectrum. This generated a wide range of hypotheses, some of which I'm still testing. For instance, it turns out that the parasympathetic side governs sexual arousal, but the sympathetic side governs orgasm. There should be something in this space that allows one to use sexuality to hack mindfulness. (This might explain what tantra was trying to accomplish.)
However, in general I agree with you. Internal motive structures are usually much more useful to understand in my experience than neural structures are. As you rightly point out, the layer of complexity at which we intervene seems to be at thoughts and feelings, not at neurology.
...but if we can pinpoint some neural structure that does this background thinking, and we know roughly how it interacts with the autonomic nervous system and possibly some other things, that could result in something insanely useful such as using body posture to massively accelerate the rate of problem incubation.
Speaking of mathematicians and dreams, I found a hilarious quote some time ago:
"Once in my life I had a mathematical dream which proved correct. I was twenty years old. I thought, my God, this is wonderful, I won't have to work, it will all come in dreams! But it never happened again."
--Stanislaw Ulam; January 14, 1974, in "Conversations with Gian-Carlo Rota"; as quoted on pg262 of Turing's Cathedral (2012) by George Dyson
I agree this is a factor in problem-solving. I've found it to be important too. However, I suspect this can't be the main reason behind the "magic genie" phenomenon because if it were, you'd expect that mind-quieting meditations between bursts of mathematical effort would be vastly more productive than spending hours on problems and taking breaks. E.g., spending fifteen minutes on a tough problem and then spending five minutes meditating, cycled three times, would produce vastly better results than thinking about the problem for an hour. I'm not aware that this is the case, nor that the 20% most prodigious mathematicians are above average in their interest in mindfulness.
I agree that it's not the main thing, but not with your analysis. For one, this "mindfulness" thing is never really unpacked well. It could be that the habit of not focusing and bouncing between ideas (that's considered "unmindful", right?) is what it takes to not get stuck in ruts, and that the helpful mindfulness related bit is meta-awareness that "I'm noticing that I'm stuck" - and then fixing it instead of freaking out about it.
Instead of practicing mindfulness by itself, I'd hold mindfulness as an ideal and attack the specific blocks more directly.
Oddly, it turns out that most mathematicians do not relate to the supposed experience of a solution presenting itself in a dream. Many, including Hadamard, have mentioned waking up to have the answer to a problem they've been working on trumpeted into mind, but not clearly as a result of dreamed experiences
This actually doesn't change my hypothesis much. I'm hypothesizing something that happens without requiring awareness. I occasionally notice myself making strange metaphorical connections that were there outside my awareness for some time and finally got bumped in. I very much expect this to happen completely outside awareness a lot. Heck, Milton Erickson was famous for doing this on purpose as a technique in therapy!
By and large, no. But in this domain, yes! [...]what I was talking about seemed to be the division between the sympathetic and parasympathetic nervous systems.
Which domain exactly?
Interesting. I'll have to look into that distinction more, and generally spend more time in that perspective. I have gotten similar benefits, just on a fairly small scale - it'd just allow me to make sense of things there were a bit elusive and import it back to the individual thoughts/feelings level model.
Alas, no. But you are certainly not the first person to ask! I get this request on almost a weekly basis. I'm tempted to make a blog just for the social acclaim, but given the opportunity cost I'll hold off until I can see a way to leverage something like that into something more clearly tied into world-saving.
I don't have much help on the neuro end, but my go-to explanation of this hasn't been assigning tasks to the homunculus, so here's what inspiration feels like to me, and if you can draw anything useful out of it, mazel tov!
The kinds of problems that feel like they are frequently resolved in this way for me include math proofs (esp linear algebra, topology, fractal geometry, for reasons that may become obvious), theses, and speeches I give or blog posts I write. In college, I was part of a philosophical debating group, and I pretty much never wrote speeches in advance. I'd think about the topic during the week, listen to speeches during the debate, and then have some BLAM moment during the debate, where I suddenly could feel the logic of my speech, and had a sense of the shape of it and where certain quotes/anecdotes/studies would fit in. Then I was ready to take the floor.
Here's what my mental map of this process is: I start thinking about a problem by thinking intensively about particular aspects of it -- these are nodes. I start building up references, linked ideas, objections, difficulties around each of these nodes. So, basically, each node starts becoming a more densely connected subgraph of (what will hopefully be) an eventual solution graph. During my active thinking, I keep taking walks in my mindspace from these nodes, hoping to find a path to link up these subgraphs. Sometimes things resolve here.
When they don't, when the problem isn't actively before me, I think of the subgraphs as still accreting ideas and connections. But now it's more of a random walk than when I'm directing it and looking for a particular connection. I imagine that when I call up some specific idea (a song lyric, a subgraph for a different problem, a passage in a book) the near connections of it are weakly activated, and, if they're close enough to one of the subgraphs I've been thinking about recently, they get aggregated on, and extend the graphs out in serendipitous ways. If the new thing I'm thinking about (or any of the things it's weakly linked to) links up two subgraphs, or puts me near enough to spot the connection, that's what inspiration feels like.
And sometimes, a weakly lit up idea prompts inspiration by being grotesque. (By grotesque, I mean close-but-not-close-enough to the right thing to give you a wiggins, and draw your attention to the particular way it falls short of what you need). The uncanny valley is for more than just robots!
Here are some examples from fiction that feel similar to my subjective experience of inspiration, at least in some particular:
Basically, thinking about a problem is building a big graph, but interacting with any of the other graph-thoughts I think about in day to day life might light up a connection to a node of interest. I find the more SNS-kinda mode more helpful here (I'm in the same kind of mood at debate as I am at improv, resting lightly on things, curious about where they'll go).
My own experience, and much of what I've read, suggests a role of dreams and sleep in insight-generation. Perhaps you solve the problem while sleeping, but your brain doesn't store the memory of figuring it out in detail because you're asleep, and the flash of insight you feel in the shower is just remembering what you already figured out.
The little man could just be the promise to yourself to sleep on it, to literally work on the problem while asleep. And you need to walk away from the problem because conscious, deliberative approaches to problem solving will trample the delicate memory of the solution you arrived at while sleeping.
That's certainly plausible! I don't think it can be tied strictly to sleep because walking away from the problem for a few hours is often good enough. Similarly, suddenly remembering someone's name at a party can happen just a few minutes after giving up, while you're still at the party. (Or at least I've personally experienced this! And I think I recall others having experienced this too.)
But there might be something near here too. For instance, I've been told that the REM rhythms of 90 minutes actually continue throughout the day as well as at night while you're sleeping. I've never tracked down this claim to evaluate it, but if it's true it might suggest that there are particular times of day when this processing happens automatically in the background and other times when the processing needs to happen more consciously, roughly on 90-minute cycles. I'm speculating pretty wildly here though.
From what I've learned about brains, the left brain is engaged in symbolic thinking about a problem, which engages more logical, methodical problem-solving. For a combination that you won't arrive at through that approach, you have to give your brain, apparently involving activation of the right dorsolateral prefrontal cortex and other right-brain parts, more time to integrate info from stored memory or lower-level processed stimuli or to make novel associations related to the problem. When left prefrontal cortex is engaged in focusing on performing a task, it'll inhibit the processing of info seemingly irrelevant to the task. This is why aha/eureka moments are more likely when you're relaxed, not focused and your mind gets to wander (e.g. getting on bus while on vacation, taking a shower/bath). Studies suggest that more creative or sudden-insight (as opposed to deliberately trying different combinations) problem-solvers have greater right brain activity and lower inhibition of it.
Look up "Aha! moments" in the index of Eric Kandel's book, The Age of Insight, which cites many papers, incl. "Explaining and inducing savant skills: privileged access to lower level, less-processed information". A few of my other references: "Bayes for Schizophrenics: Reasoning in Delusional Disorders", "Creativity tied to mental illness", "Through the Wormhole: Creativity Cap"
I think the answer is hiding in this bit: "In this particular case it also came with a feeling of total confidence that verification would pan out (although Hadamard notes that the validation step after the insight is still essential because sometimes that feeling of total confidence is mistaken)."
I'm going to guess that the class of problems this work on is where the person knows somehow that there IS an answer. Almost as if they see the pattern from the outside yet haven't just worked out the details. This would explain why when they get the answer they 'know' it's correct. It would also explain the hard work AND the frustration of not getting it. After all, if you didn't intuit that there was an answer, if on some level you didn't see it, well then you wouldn't feel frustrated at not getting it!
And so the question then is, if you know the answer, if you can see it, well then why the heck can't you know it?!
For that I think that sometime it takes some time to fully experience what you see and know. For it to open up completely from implicit body knowledge to explicit cognition.
Now if you don't think about it at all, when then it's dormant. It's not on the table. At the same time, if you work on it too hard you very very often block yourself by 'going too fast' with your attempts to 'get it out there already'.
When you work on it, then stop, you have put it on the table and at the same time you aren't getting yourself stuck in a rut by trying too hard.
As for what is actually happening when you aren't thinking about it, well at all times you're implicitly holding in mind many many things. Most are invisible to you (the fact that you live in country X and have family Y and have profession Z). You act on those implicit contexts, you must be 'making contact' with them, yet they are invisible (do the fish know of the water they swim in).
Added to that are the implicit stuck things in your life. Like knowing that the whole thing with the landlord is not resolved... Even when you aren't thinking about it, you feel it. You sense it. Walking around during the day with that issue 'on the table' compared to walking around without it is a completely different experience.
A good way to experience this experientially in less then 5 minutes is to ask yourself "What is not perfect in my life?" When you get the answer, write it down and then say "And X isn't perfect, and without X, what's not perfect in my life?" Keep going till it stops. You'll know when to end... :)
When you do that exercise you FEEL the body sense of layers on implicit stuff that you're holding come off, if even for a moments.
When you have a difficult problem that you KNOW is solvable and you can't find it, well you feel that. All day. And that's the implicit processing going on.
(Let us know how the exercise went).
The literature calls it "incubation". Happy digging.
I'm currently testing a promising direction for a possible collection of units at CFAR. (For those who have attended some CFAR events or test sessions, this is a collection of refinements to the fudoshin/"panic" unit.) I've hit on what I think is a key puzzle whose answer might unlock a lot of the emerging art of rationality. I - and possibly most people here, eventually - would very much appreciate any insight you have to share.
The puzzle is how thought incubation works, ideally expressed in terms of neural systems or neuroanatomical structures. I'll first explain the phenomenon and then suggest the general reference class from which I'm hoping to get an answer.
The Phenomenon:
Mathematicians frequently report that often one of the most helpful things they can do to solve a problem they're stuck on is step away from it. Jacques Hadamard (1949) examined his own experiences and also talked to many of his colleagues to work out what the common structure of this experience was, and determined that there seems to be a fairly predictable sequence to it:
(1) Intensely focus on the problem, working through every permutation you can think of that's likely to produce an answer.
(2) Walk away from the problem and think about something else.
(3) The magic genie in your head might eventually, and often unexpectedly, yell a possible insight into your awareness.
For instance, Henri Poincaré reported struggling to work on Fuchsian functions over the course of several weeks and then being forced to walk away from the proof he had been stuck on due to a planned vacation. One day he was stepping onto a bus with his mind certainly not on mathematics, and suddenly the key insight he needed to finish the proof appeared in his mind. It was as though a part of his mind had been secretly working on the problem and then brought the finished product into his awareness. In this particular case it also came with a feeling of total confidence that verification would pan out (although Hadamard notes that the validation step after the insight is still essential because sometimes that feeling of total confidence is mistaken).
I definitely relate to this from when I was working on graduate mathematics. However, it also pattern-matches with other mental phenomena that are much more common. For instance, sometimes I think I know what a person's name is, but struggle as I might I can't quite remember it - and then a few minutes later after I've given up remembering the name the answer loudly announces itself, often quite out-of-context. Or when I'm trying to figure out a way of improving a throw in martial arts and then find the answer suddenly dawning on me at a random time.
I'm under the impression that this is a fairly universal kind of experience. I suspect you can think of examples in your own life where this has happened. ("Oh, now I remember where I put those keys!")
Reference Class for an Explanation:
I'm going to offer some overly simplistic examples of the kind of explanation I'm looking for. In this case, I think overly simplistic might be okay since I'm just trying to get a reasonable handle on how to munchkin the interaction between a few different neural systems. If it turns out that a more detailed and technically correct version is important, I'll probably dig into it (pending the VOI versus cost-of-information comparison).
There seems to be some evidence that one of the reasons children are as impulsive as they are is that they haven't yet developed their prefrontal cortices (PFCs) to the degree adults have. The prefrontal cortex seems to do at least two things: (1) hold long-term goals in mind and (2) engage executive function (i.e., halt orders on impulses, typically ones that don't match up with the long-term goals). This neuroanatomical structure seems to continue growing until sometime in one's early 20s - which might be why we also find that teenagers typically have less impulse control than twentysomethings but more than middle-schoolers, whereas we don't find such a clear distinction between twentysomethings and thirtysomethings. (Yes, this could also or even instead be cultural. I know it's complicated.) Incidentally, I understand that the PFC is also one of the neural structures most deactivated by alcohol - although my impression is that it shuts down the long-term goals thing and not the executive function. (This is based on my and others' experience that precommitment works perfectly well. It seems to me that saying things like "I couldn't help myself because I was drunk!" is more a social excuse than an actual explanation. But I'm only around 65% confident of this as a general claim.)
On a related note, it would seem that there's something in the same rough space as theory of mind that goes beyond the ability to pass the false-belief test. According to Rebecca Saxe, the capacity for empathy seems to come from a particular bit of the brain that doesn't finish growing until the mid-20s. Saxe also provides some evidence that a sufficiently strong and precisely directed magnet can basically deactivate that part of one's theory of mind temporarily. It seems quite plausible to me (though I really don't know) that activation of the sympathetic nervous system (SNS), such as in fight-or-flight reactions, decreases activation of this empathy part of the brain. This might be why, in a perceived crisis, some people switch to an almost tool-like view of others (e.g., knowing that overcoming the bystander effect requires pointing at a specific person and saying "You! Call 911!" but not really getting a sense in that moment of what that person's experience is like to be so singled out).
I'm quite aware that much of the above is speculation. I think speculation is fine, but having it grounded in some actual known neuroscience is ideal. That would give me something to dig into. E.g., if there's some reason to believe that this phenomenon is related to the enteric nervous system, I can start digging into the literature on that system to better understand how to munchkin its interactions with the (rest of the) autonomic nervous system.
An example of something outside the reference class I'm looking for is a "little man in the subconscious" explanation. I first read about this about twenty years ago as a model for how mental incubation works: you concentrate on a problem in order to communicate to a little man in your subconscious what you want to have done, and then you stop talking to him so he can go do what you just told him to do. Then he comes back with an answer once he's done, without regard to what you're doing when he's done. I agree that this seems to be a reasonable metaphor for what's going on, but it doesn't tell me for instance why the "little man" seems to respond so much more to SNS activity than parasympathetic activity, or why he can't go do his job once he has the instructions even if we continue to think about the problem.
More generally, psychodynamic "explanations" are unlikely to be helpful here. Talking about this as the "domain of the iNtuiting function" in reference to Jungian psychodynamic theory or Myers-Briggs won't tell me hardly anything about how this relates to stress oscillation.
So... Any suggestions about what this mysterious "little man" might actually be made of?