Are you familiar with the concept of fold/unfold? Folds are functions that consume structures and produce values, while unfolds do the opposite. The composition of an unfold plus a fold is called a hylomorphism, of which the factorial is a perfect example: the unfold creates a list from 1 to n, the fold multiplies together the entire list. Your section on the "two-fold recursion" is a perfect description of a hylomorphism: you take a goal, unfold it into a plan composed of a list of micro-steps, then you fold it by executing each one of the micro-steps in order.
Wow, that's really cool to learn! I only have a beginner level knowledge of functional programming concepts and was not aware of hylomorphisms and unfolds (just basics like fold left, fold right). Thanks for bringing that to my attention, I might try to read that whole series.
The post was very clear to understand, and the graphics were spot-on (would even be fine to leave just the graphic trees without the text trees).
I actually done something very similar to that a few days ago. I wrote down on a peace of paper a few things i knew are important to me, and than analyzed them - thought how do they relate to each other, what might be higher than them, and how might i break them down. than i also connected each breakdown to other higher level goals that could be benefited from achieving it.
After doing it i got to a conclusion that there are two shorter-term goals that are most beneficial for me right now.
This is basically a 'technical' explanation of the "start with the end in mind" principle, which i try to use often. hopefully you gave me another way to think about it which will be useful.
Really enjoyed this. I've found myself using this concept a few times in my thoughts just the past couple days since I read this.
One more level of recursion: I need to plan some time-allocation for prepare the actually good plan of action. For example, before travelling, I sometimes spent a few days planning my trip, so I should reserve time for planning.
Trip plan:
1) 2 days detailed trip planning
2) Buy tickets
3) etc....
This also should be included into subplans, as during the trip I could have new data and may beed to spend more time in the beginning of each substep to plan it.
For complex tasks (like medical scientific experiment) recursive meta-planning becomes probably most time-consuming part, especially if includes coordination between many agents.
This could create problems, in style of Buridan ass, as it is impossible predict the time for planning before planning.
Obvious AI connection: goal encapsulation between humans relies on commonalities, such as mental frameworks and terminal goals. These commonalities probably won’t hold for AI: unless it’s an emulation, it will think very differently from humans, and relying on terminal agreements doesn’t work to ground terminal agreement in the first place. Therefore, we should expect it to be very hard to encapsulate goals to an AI.
(Tool AI and Agent AI approaches suffer differently from this difficulty. Agents will be hard to terminally align, but once we’ve done that, we can rely on terminal agreement to flesh out plans. Tools can’t use recursive trust like that, so they’ll need to explicitly understand more instrumental goals.)
Seems like the planning process or algorithm is recursive but the plans are merely hierarchical.
Speaking or recursion in human cognition, I've always wondered if it is implemented in the human brain in what computer scientists (programming language compiler writers, to be specific) call "unrolling" as opposed to true recursion. Many modern compilers, when they detect that a recursive algorithm or simple loop will actually only nest 5 times, say, will generate machine code that unrolls the recursion into a simple linear series of 5 steps. The brain really can't handle very many levels of recursion so this may be why. it implements what abstractly requires recursion as a linear sequence, turning the recursion level into a simple sequence index. Nature never (or hardly ever) implements true recursion as it always stops after a few levels.
The brain really can't handle very many levels of recursion
Nature never (or hardly ever) implements true recursion as it always stops after a few levels.
Do you have a source for these? (a book on neuroscience, a picture of simple animal brains, etc.)
Epistemic status: Reference. Highly confident. I rely on what is presented here extensively in my own thinking.
Plans are recursive. Any plan can be decomposed into parts and those parts can be in turn be decomposed into further parts and so on until it is senseless to decompose any further. This is not a profound point, it is something we all appreciate intuitively and explicitly to at least some degree or another.
Nonetheless, the recursive nature is of such fundamental importance to the entire practice of planning that it warrants an explicit treatment.
A quick definition of planning as used in this post: Planning is the selection of actions to be deliberately executed in order to achieve a desired outcome/goal. [1]
Plans are recursive
Usually, we treat plans in a very linear fashion. Step 1, Step 2, Step 3. To-Do lists capture this, and we can also draw an accompany thing diagram.
Example Linear Plan: Hosting a Dinner Party
Linear diagram of a plan to host a dinner party
Of course, we all recognize that the three steps of the plan above are composed of further sub-steps. Nested to-do lists capture this well.
Example Recursive Plan: Hosting a Dinner Plan Expanded
These steps could be further decomposed, but this suffices to demonstrate the idea. We can draw another diagram.
Recursive tree diagram of a plan to host a dinner party
Why call it recursion?
Sometimes people use fancy technical terms in order to sound smart even though a more commonplace word would suffice. Arguably, I could have said that plans are hierarchical or nested or something. Why use the mathematics and computer science term, recursion?
I defend this choice with reference to the defining features of recursion:
[The calculation of factorials is the canonical introductory example for recursion and neatly illustrates all of these features.]
Planning has all of these features and is therefore rightly described as being recursive.
We see that plans can be decomposed into sub-plans which might then be decomposed into further sub-plans. Eventually you stop, i.e. you reach a base case. When is this? It depends on the context, but one heuristic is to stop when the level is so trivial that it doesn’t require further decomposition. In the above example of hosting a dinner party, I might not need to recurse to the level of planning out the steps of sending an emails. I can take for granted that I know how to carry that out.
The Core Planning Process
What is the universal planning function/procedure in the context of planning? A decision theorist or AI expert might be able to offer a little more rigor here, but the universal core planning process we repeatedly apply whenever we are planning is going to look roughly like this:
How the human brain executes the above steps is complicated. And what is required to execute those steps will also differ dramatically between plans and at different levels within a plan. In particular, sub-plans can require very different models to map actions to outcomes than the models used at a higher level. The models used for business strategy are different from those used to file taxes, though both are part of the broader “business success” goal. Still, we shouldn’t let the variation between plans mask that the core steps are always the same.
Plans are Two-Fold Recursive
We can actually say that planning is doubly recursive. Planning is both recursive in planning, i.e. when you’re choosing what to do [2], and also in execution. To execute a plan, you must execute each of each the sub-plans, and each of the sub-plans’ sub-plans and so on. When we’re talking about execution, the recursive function to be applied at each level is simply “Do X”.
Implications of the Recursive Nature
The following section details some key principles of good planning that can be directly derived from the recursive nature of plans. Even though both the recursive nature and the principles presented here seem sensible, if not obvious, on their own, linking the two together makes it clear why these planning principles are necessarily so.
Goals/Steps/Actions/Plans are Isomorphic
When we plan, we usually start with a goal. Then we figure out actions and sub-plans we will take towards the goal. This picture is somewhat misleading. It makes it sound like the goal and steps we take towards it are of a different kind, and so encourages us to treat them differently. Usually, that means treating the goal as more fixed than it ought to be.
Someone might decide that they have the goal of becoming a lawyer. They start of with this goal and start evaluating steps they might take towards achieving it. Maybe they start thinking about which law school they want to go, where they will get a student loan, how they’re going to improve their study habits. They start considering the obstacles and how to overcome them.
For each possible step towards the goal, they enumerate and evaluate the options for their benefits, costs, and risks. What is so easily missed is whether or not the “goal” was the right step to begin with. One must remember that the goal itself is likely only a step within some yet larger step. The goal is a step too because, from the perspective of the recursive tree, all the nodes are of the same type. [Perhaps we wish to treat nodes at the very top or very bottom differently, but otherwise, there’s no difference.]
It is only in our minds that we distinguish between goal, step, action, and plan. Structurally they are the same thing.
See? They’re all just plans.
The exception to the isomorphism are terminal goals which we wish to accomplish purely for their own sake rather than instrumental goals which we seek to accomplish to attain something else. Terminal goals are not steps within another plan and so the isomorphism does not hold with them. Still, I claim that since the overwhelming majority of our goals and plans are only instrumental, it is reasonable to assert that goals (generically) are isomorphic to plans. Most nodes in the recursive tree are in the middle. It’s true enough that it’s helpful to remember it as such.
Almost All Plans are Sub-Plans; Almost All Goals are Sub-Goals
The corollary to the above point that goals/steps/actions/plans are all the same is that [almost] all plans are sub-plans and [almost] all goals are sub-goals. If you have a goal, it’s probably because there’s something you hope that goal will accomplish.
We can reuse the lawyer example from above.
What can be lost when you start with your goal and don’t look any higher is all the reasons you selected your goal in the first place. It is worth remembering and examining those reasons. Possibly you selected those reasons long ago and they no longer apply. Possibly it was an intuitive or emotional decision but not an especially good one. If you don’t think about the reasons behind your goals, i.e., your higher-level goals, then you very likely will either select the wrong goals or select the right approximate goal which you then execute in such a way as to not accomplish your higher-level goals.
Peter Thiel, famous billionaire investor, went to Stanford law school and practiced in a prestigious New York law firm for seven months and three days. He is oft quoted as describing that this law firm was a place where “from the outside everyone wanted to get in and on the inside everybody wanted to get outside.” When he left, a colleague told Thiel that he didn’t know it was "possible to leave Alcatraz." Clearly not somewhere he actually wanted to be. Peter Thiel cites his reason for going to law school in the first place: "So I think I got a JD because of the abstract prestige, because I didn't know what else to do, and I didn't concretely think through what it would be like to be a lawyer."
Peter Thiel’s example highlights an important point. If you know why you have a particular goal, you can list out the assumptions and beliefs that cause you to have it. Once listed, the assumptions and beliefs can be examined, further evidence can be collected, and you can avoid making plans based on shoddy beliefs and limited information. You can ensure that you think through concretely what it would be like to be a lawyer, or as Thiel advises others, go identify someone with a JD who’s a good role model.
The elucidation and examination of assumptions should be part of your “core planning process” which should carried out appropriately at each level and node of your plan.
In a sense, you need to dissolve the distinction between plan/goal/action/step and see the recursive tree of your plan for what it is. Then you apply your planning process to the nodes indiscriminately. When starting with a goal, you ought to “recurse upwards” until you identify higher-level goals that you are sufficiently confident in [3]. Even if you are completely certain about your higher-level goals, you still want to be thinking about them because their details dictate what the details of your lower-level goals need to be.
When our lawyer-wannabe sits down to plan how they’re going to become a lawyer, they will make worse choices if they forget what the point of becoming a lawyer was. The point of a lawyer is not to become a lawyer inherently, more likely it’s to have a good job with a high salary, prestige, and satisfying work as part of overall having a good life. You can avoid repeating Peter Thiel’s mistake.
I fear that the very particular obvious career example I have presented won’t translate readily in the minds of readers to all the other cases where this applies. It applies to projects within startups and business, it applies to the books you read, it applies to the people you date and how you find them, it applies to your hobbies and your vacation. It applies to any planning of non-trivial complexity. I encourage the reader to search for a few examples from their own life where they might benefit from applying this line of thinking.
A goal is only right if it actually serves your higher-level goals
If you didn’t pick the right goal to begin with, then no matter how hard you hit that goal, all is for naught. In terms of a recursive trees, for a goal to matter, there needs to be an “unbroken path” to something you terminally value.
If the lawyer-wannabe succeeds super hard at becoming a lawyer: graduates Summa Cum Laude from the best school, gets a role in the most prestigious firm, etc., it doesn’t matter at all if it turns out being a lawyer was the wrong choice.
To reword points from above, goals can be wrong in two ways:
An aside about long-term planning and feedback
High-level goal selection is often fraught because there is slow feedback. A young person might not discover for years that they didn’t actually want to be a lawyer, so it’s easy to pick the wrong high-level goal and grind towards for years, only getting feedback that it was the wrong choice after they graduate and start working. Peter Thiel is an example. Another class of examples is startups and business deciding which products to build. Done wrong, a startup can pick a long-term goal and work for months or years before finding out it was the wrong goal.
Worse than the pure absence of quick-feedback, long-term planning is often influenced by short-term factors such as emotions. “Gah, planning is stressful! I’m just going to pick something and stick with it.” Rather than focus on the long-term goal, how the person feels in the moment is prioritized. Relatedly, as with Peter Thiel, instead of trying to pick the best long-term plan, someone might pick the plan which gets the greatest social reward (since there are indeed rewards for just having cool plans). There can faster feedback on whether your plan is cool then whether it will succeed. You’ll know as soon as you boast to your friends or describe your product plans to your startup's investors. The quick feedback on emotions and social rewards often outweighs the value of a long-term goal one is supposedly planning towards. Beware.
The precise success criteria of a lower-level goal are determined by the higher-level goals
The importance of being clear on why a goal has been selected is never clearer than in the case of delegation. A manager asks a research engineer to write a report on their recent research. The engineer does so, but the manager is not pleased! The report is a technical, suitable only for other engineers, yet this is not what the manager wanted - the manager wanted a report to give to the business executives.
The research engineer delivered what was requested “a report on their recent research”, but since this didn’t connect with the higher level goal of having a way to update the executives, it was worthless.
The danger is that we humans often encapsulate our goals and plans very crudely and with very little detail. A few words like “become a lawyer.” We rely on ourselves and others implicitly understanding the specific versions of those goals which would count as success - yet so often we don’t.
Even when the goal is certain, you need to understand the bigger picture in order to be able to make lower-level implementation choices. Anyone delegating a task is advised to instruct the delegatee in why the task is being requested. The more detail the better.
You can’t just ask someone to book you a venue for an event. You either need to tell the exact kind of venue you want or tell them what kind of event it is so they know what kind is appropriate. And even if you say “fits 50 people, within X budget, within Y miles of location A” there could still be further decisions whose ideal selection is made only if you know why a venue is needed: what kind of event, which guests and therefore which ambiance and so on would best.
Delegation is hard because it means splitting the recursive tree among multiple people. In the case of principal-agent problems, you’re transplanting a goal from one person’s tree to another. You can see why that’d run into trouble.
Final example: as a Data Scientist, I was often asked to build things with little specification and no explanation of why. A technical example here is that if you don’t understand how a classifier will be used, then even if you can build one, you don’t know how to make a dozen relevant design choices, e.g. setting thresholds which determine false positives and false negatives. Building something without an idea of why is a great recipe for building a version of it which doesn't meet the true user requirements. [4]
Wrapping up
Though the ideas presented here are simple, I hope that if you have read this far, this post will have deepened your appreciation of this fundamental property of plans and will result in you making better plans towards good things.
Appendix: Multiple Goals
Though it was skipped for the sake of simplicity in the main text, realistically, many our plans are executed with more than one goal in mind. This is entirely compatible with a recursive model of plans, but it makes the diagrams messier. In order to not distract from the main points, examples in the text were chosen to roll-up neatly into a single goal.
Below is a realistic recursive plan diagram illustrating multiple goals. Makes you appreciate what your brain has to deal with.
Modern art: a recursive tree diagram of a plan where sub-plans feed into multiple higher-level plans, e.g. actions are taken towards multiple goals.
Endnotes
[1] This broad definition stands in opposition to a common definition which uses planning primarily in contexts of scheduling, e.g. plan your day, plan your week. The broader definition here is essentially synonymous with decision-making, perhaps differing only in connotation. Decisions somewhat imply a one-off choice between options whereas planning implies selection of multiple actions to be taken over time. The term planning also somewhat more than decision-making highlights that there is a goal one wishes to achieve.
[2] With the broad definition of planning used here, it’s important to note that planning needn’t happen all at once. More commonly, different components of a plan (selection of action) happen at different times. Very commonly, higher-levels of the plans are selected up front and lower-level actions are selected near or at the time of execution.
[3] This is often really, very hard and time-consuming. If you’re haven’t recursed explicitly upwards to determine what you want and what your true high-level goals are, this might take a while including both the time to do it and possibly the time spent gaining the skills do it. The good news is that after initial investment you get to reuse your knowledge of your high-level goals repeatedly when making lower-level plans such that you make much better low-level plans.
[4] My boast: determined to not build the wrong thing, I spent enough time figuring out what the why reasons for our engineering work should be that my company eventually asked me to take that over as my explicit role.