Pretty sure I need to reverse the advice on this one. Thanks for including the reminder to do so!
I really like this. I think it should indeed apply equally to a startup-like context such as LessWrong. We already periodically have strategy retreats, but this "drilling down along a new and different branch of the tree" framing, that's not an approach I had before. Thanks for writing this up.
I use an alternative technique that works well for me - making sure to walk up the stack on every significant new development at lower levels.
E.g. if on level 5 am trying to solve X with technique Y, and I realize that it does not quite work, but I would probably be able to do X' that is as good with Y', before jumping into Y', I take time to consider - well, X' is as good as X for level 4, but does it perhaps mutate level 4 away from higher-level goals? Maybe the fact that Y does not actually work for X indicates that approach at one of the higher levels is off?
And it's actually similar when Y does succeed for X - once it does, I learned something new, and need to check my stack again. Or maybe I realize that Y is taking me much longer than expected - again, need to walk the stack and figure out whether X and Y are even worth it. This way when I am in the zone on Y, there is no distraction, but I also do not have the stack ignored for too long as beeing in the zone for Y for too long is an indication that something went wrong and the plan needs to be reexamined.
Having hard deadlines, even artificially imposed, helps. Having goals explicit (and explicitly written, so that I can remind yourself how I ended up in the rabbit hole I am in) for each of higher levels helps.
YMMV, of course.
My mileage varies. I have a bias for the 5th level, and if I'm currently deeply immersed in a rabbit hole that I reflectively think is usefwl, then going up the ladder again risks reminding me how distal the rabbit hole objectives are. I remind myself just how much I care about saving the world, but the caring mostly leaks out when trying to reach more distal instrumental objectives.
The "drilling down along a new and different branch of the tree" concept makes me think of tree search algorithms, naively being depth or breadth first searching. It's overly simplified, but might uncover related theory.
The goal is to search from whichever node you estimate to being closest to the goal. Calculating the estimate is difficult, so we tend to only look at a small nearby neighbourhood, which is usually low level. Backtracking forces you to make estimates for earlier nodes.
If I was making this algorithm faster, I'd try to find a way to make the heuristic (the estimate of nearness to the goal) more efficient. I've no idea how to do that, but maybe looking at how past discoveries were made could help.
Then again, given that research takes a long time, maybe it's not worth making any sacrifices to the heuristic accuracy.
(This is about a research productivity strategy that’s been working very well for me personally. But YMMV, consider reversing any advice, etc. etc.)
As a researcher, there’s kinda a stack of "what I'm trying to do", from the biggest picture to the most microscopic task. Like here's a typical "stack trace" of what I might be doing on a random morning:
So as researchers, we face a practical question: How do we allocate our time between the different levels of the stack? If we’re 100% at the bottom level, we run a distinct risk of "losing the plot", and working on things that won't actually help advance the higher levels. If we’re 100% at the top level, with our head way up in the clouds, never drilling down into details, then we’re probably not learning anything or making any progress.
Obviously, you want a balance.
And I've found that striking that balance properly isn't something that takes care of itself by default. Instead, my default is to spend too much time at the bottom of the stack and not enough time higher up.
So to counteract that tendency, I have for many months now had a practice of "Solve The Whole Problem Day". That's one day a week (typically Friday) where I force myself to take a break from whatever detailed things I would otherwise be working on, and instead I fly up towards the top of the stack, and try to see what I'm missing, question my assumptions, find new avenues to explore, etc.
In my case, "The Whole Problem" = "The Whole Safe & Beneficial AGI Problem". For you, it might be The Whole Climate Change Problem, or The Whole Animal Suffering Problem, or The Whole Becoming A Billionaire Problem, or whatever. (If it's not obvious how to fill in the blank, well then you especially need a Solve The Whole Problem Day! And maybe start here & here & here.)
Implementation details
Why do I need to force myself to do this, psychologically?
It's crazy: practically every Solve The Whole Problem Day, I start the morning with a feeling of dread and annoyance and strong temptation to skip it this week. And I end the day feeling really delighted about all the great things I got done. Why the annoyance and dread? Introspectively, I think there are a few things going on in my mind:
But again that's just me. YMMV.
(Related: Richard Hamming's "Great Thoughts Time" on each Friday afternoon.)