I think some equivalent of Steam pops out as an important concept in enabling-agency-via-determinism (or requiredism, as Eliezer calls it), when you have in your universe both:
The latter is something that can also become a very solid (full of Steam) thing to lean on for your choice-making, and that's an especially useful model to apply to your selves across time or to a community trying to self-organize. It seems very neglected, formally speaking. Economically-minded thinking tends to somewhat respect it as a static assumption, but not so much the dynamics of formation AFAIK (and so dynamic Steam is a pretty good metaphor).
However, shouldn't "things that have faded into the background" be the other kind of trivial, ie. have "maximal Steam" rather than have "no Steam"? It's like an action that will definitely take place. Something that will be in full force. Trivially common knowledge. You yourself seem to point at it with "Something with a ton of steam feels inevitable", but I suppose that's more like the converse.
(EDIT: Or at least something like that. If a post on the forum has become internalized by the community, a new comment on it won't get a lot of engagement, which fits with "losing steam" after it becomes "solid". But even if we want to distinguish where the action is currently, it makes sense to have a separate notion of what's finished and can easily re-enter attention compared to what was never started.)
Also when you say, in your sunk costs example, "no steam to spend time thinking", I'd say a better interpretation than "time thinking" would be "not enough self-trust to repledge solidity in a new direction". Time to think sounds to me more like Slack, but maybe I'm confused.
when you have in your universe both:
Indeed, this seems quite central.
However, shouldn't "things that have faded into the background" be the other kind of trivial, ie. have "maximal Steam" rather than have "no Steam"?
I agree that this is something to poke at to try to improve the concepts I've suggested.
My intuition is that steam flows from the "free-to-allocate" pile, to specific tasks, and from there to making-things-be-the-case in the world.
So having lots of steam in the "free-to-allocate" pile is actually having lots of slack; the agent has not set up binding constraints on itself yet.
Having lots of steam on a specific task is having no slack; you've set up constraints that are now binding you, but the task is still very much in the foreground. You are still often trying to figure out how to make something happen. However, parts of the task have become background assumptions; your attention will not be on "why am I doing this" or other questions like that.
Finally, when steam flows out to the world, and the task passes out of our attention, the consequences (the things we were trying to achieve) become background assumptions.
... Or something like that.
I'm getting some sort of "steam = heat" vibe from this. You apply steam to heat a situation up until it melts and can be remolded in a new form. Then you relax the steam and it cools and solidifies and becomes part of the background.
More generally it's like energy or work. Energy is the ability to push against a given force a given distance - to overcome inertia / viscosity and modify the state of the world. After that inertia keeps the world state the same until something else changes it. Perhaps viscosity - probably the wrong term, but I mean the amount of pushback if you try to make a change to worldstate, which might vary depending on the "direction" you want to push things - is also a quantity worth thinking about?
Ooh! More generally, energy is about accelerating a mass through a distance. But momentum remains. Perhaps a way of doing things that is stable has lost steam (acceleration) but retains high momentum?
Thanks for clarifying! And for the excellent post :)
Finally, when steam flows out to the world, and the task passes out of our attention, the consequences (the things we were trying to achieve) become background assumptions.
To the extent that Steam-in-use is a kind of useful certainty about the future, I'd expect "background assumptions" to become an important primitive that interacts in this arena as well, given that it's a useful certainty about the present. I realize that's possibly already implicit in your writing when you say figure/ground.
I love this post. (Somehow only just read it.)
My fav part:
> In the context of quantilization, we apply limited steam to projects to protect ourselves from Goodhart. "Full steam" is classically rational, but we do not always want that. We might even conjecture that we never want that.
To elaborate a bit:
It seems to me that when I let projects pull me insofar as they pull me, and when I find a thing that is interesting enough that it naturally "gains steam" in my head, it somehow increases the extent to which I am locally immune from Goodhardt (e.g., my actions/writing goes deeper than I might've expected). OTOH, when I try hard on a thing despite losing steam as I do it, I am more subject to Goodhardt (e.g., I complete something with the same keywords and external checksums as I thought I needed to hit, but it has less use and less depth than I might've expected given that).
I want better models of this.
This post has been surprisingly important to me, and has made me notice how I was confused around what motivation is, conceptually. I've used Steam as a concept maybe once a week, both when introspecting during meditation and when thinking about AI alignment.
I remember three different occasions where I've used steam:
I remain unsure what differentiates steam from motivation or flow, or whether steam is on the way towards mathematization of those concepts. As such, there is a pretty big relation between steam and interactions of multiple agents in a mind (gestured at with having steam work both in a societal and an "individual" context).
I really like the short definition by lukehmiles,
Steam is […] "recursive self trust with self common knowledge" or something.
I would be really interested in seeing Steam be formalised further, and feel like it may play an important role in a "generalized vNM", which describes the relation between nearly-fulfilled vNM axioms, probabilities and Steam. As such, some more math (maybe even code?) would be very cool for this.
For the reasons detailed above, I'm giving this post a +9, with hope for more.
For companies, this is something like the R&D budget. I have heard that construction companies have very little or no R&D. This suggests that construction is a "background assumption" of our society.
Or that research is happening elsewhere. Our society might not give it as much focus as it could though.
In the context of quantilization, we apply limited steam to projects to protect ourselves from Goodhart. "Full steam" is classically rational, but we do not always want that. We might even conjecture that we never want that.
So you never do anything with your full strength, because getting results is bad?
Well, by 'we' you mean both 'you' and 'a thing you are designing with quantilization'.
Anything name-able and not hopelessly vague seems to be bad to full-strength optimize. Although we should be open to exceptions to that.
As a life philosophy, it might be pretty uninspiring.
Clarification:
"Steam" is one possible opposite of Slack. I sketch a speculative view of steam as a third 'cognitive currency' almost like probability and utility.
Are 'probability' and 'utility' meant to be the other two cognitive currencies? Or is it 'Slack', and if so which is the third?
I intended the three to be probability and utility and steam, but it might make more sense to categorize things in other ways. While I still think there might be something more interesting here, I nowadays mainly think of Steam as the probability distribution over future actions and action-related concepts. This makes Steam an epistemic object, like any other belief, but with more normative/instrumental content because it's beliefs about actions, and because there will be a lot of FixDT stuff going on in such beliefs. Kickstarter / "belief-in" dynamics also seem extremely relevant.
"Steam" is one possible opposite of Slack. I sketch a speculative view of steam as a third 'cognitive currency' almost like probability and utility. This is a very informal and intuitive post, outlining concepts which could possibly have a nice formal correlate later.
This post came out of a discussion with Sahil Kulshrestha. Much of the idea is his, but this is my own take on it, especially the terminology.
Gaining Steam
When you first get an idea, it might have very little steam. It's hypothetical; exploratory. It doesn't suck up much slack.
Maybe you start idly generating plans based on your idea. It's still hypothetical, but it's starting to gain steam.
Then you notice yourself making plans. Maybe you start thinking of yourself as "making plans". At this point, the idea is starting to gain steam.
Then you might start acting on your idea. At this point, it probably has a fair amount of steam.
Full steam means you're going all-out. You have no time for anything else. All of your thoughts are directed at this one goal, at least for now. You don't have motivation problems or hesitations.
So, in an individual person, steam is something like energy/willpower/interest.
In a society, steam is something like political will. An idea outside the Overton window has very little steam -- it's hard for any political movement to work toward that idea. Ideas within the Overton window have varying amounts of steam based on the extent of their support.
Ideas like democracy, which are considered fundamental to our society, are running at very high steam.
Losing Steam
You're starting to lose steam if you are still working on something, but you're feeling a bit tired and you're not sure how much longer you're going to work on it.
An idea might lose a lot of steam if you learn about a negative consequence of that idea.
Losing steam can be an important aspect of getting things done. Losing steam helps agents solve the procrastination paradox. Roughly: if you expect yourself to do something for sure, then you can always put it off, so you might never get it done. If steam is a finite resource which you have to allocate carefully, you can't get away with procrastination like this.
Steam as Action Probability
The simplest model of "steam" is that it is the agent's own subjective credence that it will do something.
According to this, actions, policies, and plans will gain steam when you see evidence for them, and lose steam when you see evidence against.
For example, every time you resist eating cookies, you provide yourself with evidence that you are the sort of person who resists eating cookies. This might also be weak evidence that you do other healthy things like exercise.
Spurious counterfactuals and troll-bridges both involve assigning probability zero to the intuitively correct action. Perhaps we can understand those cases better if we think of those actions as being low on steam?
Steam as Optimization Pressure
A slightly more sophisticated model of "steam" is the force that moves action probabilities around. In the context of quantilization, this is an explicit part of the model: quantilizing agents have a starting probability distribution over their output/policy, and can only alter that probability distribution to a bounded extent.
"Active Inference" agents have a similar thing: a starting probability distribution which gets biased toward success. Although in that case it's harder to name a specific quantity as the "steam".
For other agents, the "initial distribution" is something like search order, and the "steam spent changing the initial distribution" is the amount of time allocated to the search.
In the context of quantilization, we apply limited steam to projects to protect ourselves from Goodhart. "Full steam" is classically rational, but we do not always want that. We might even conjecture that we never want that.
We're "putting steam into something" if we're putting a lot of time and attention into it. Especially attention. (You don't automatically get better at a task just because you practice, contrary to popular wisdom. You have to pay attention and look for ways to improve.)
For society, this is something like the spotlight of social attention. Policies get changed when enough people are trying to change them. Other things fade into the background. It's a figure/ground thing -- things we're not putting steam into are "just the way it is". The moment we start questioning, we're spending some steam in that area.
For companies, this is something like the R&D budget. I have heard that construction companies have very little or no R&D. This suggests that construction is a "background assumption" of our society.
This ties in with sunk-cost fallacy. If we've spent a lot of steam pushing ourselves in one direction, we're going to have to spend even more steam to switch directions. (The sunk cost fallacy is 100% rational if we have no steam left to spend -- no time to re-think our choices. It gets increasingly irrational as our steam budget increases.)
This also could make steam a useful commitment device. Putting a lot of steam in one direction doesn't guarantee that you'll keep those habits/policies, but it creates momentum in that direction, which helps.
Steam as Common Knowledge
One role of voting is to aggregate collective preferences. But another role of voting is to create common knowledge about a decision, so that society can coordinate on that answer. In this capacity, voting works purely because there is common knowledge that it works.
(For this purpose, a dictatorship with strict successorship rules is a similarly effective mechanism, so long as a similar portion of the society abides by it and there is common knowledge that this is the case.)
The concept of steam seems closely related to the Overton window. As things gain steam from zero, they fall further and further within the Overton window. Something with a ton of steam feels inevitable.
Steam is coordination currency.
An agent who is good at puzzles but doesn't think it's good at puzzles will avoid puzzles when there are other ways to get what it wants (because it expects to fail); will give up easily on puzzles (because it doesn't expect increased effort to result in any payoff); and when forced into a situation involving puzzles, will plan for failure rather than planning for success (EG, would willingly bet against itself). If, furthermore, the agent realizes all of these things about itself, that will only further reinforce those patterns. For most purposes, this agent is actually bad at solving puzzles.
Now imagine that this agent knows that it is good at puzzles but doesn't know that it knows. You get many of the same effects. It won't often start a puzzle, because it expects to stop before finishing. It expects its future self to avoid puzzles, so it'll make plans accordingly.
Notice how similar this is to "not trying" or "not putting in an effort". Perhaps willpower is really mostly about self-trust -- knowing what you are capable of, and knowing that you know, etc etc, so that you can coordinate with yourself.
This relates to the difference between UDT 1.0 and UDT 1.1. UDT 1.0 didn't try to coordinate with itself at all. UDT 1.1 solves the problem by optimizing the whole policy at once. (This is a computationally unrealistic coordination mechanism, however; bounded agents have to find other ways to coordinate. I have some hope that a concept like 'steam' can help describe more computationally realistic self-coordination methods.)
We normally think of agents as automatically having common knowledge with themselves, about whatever it is they know. Yet, bounded agents will inevitably fail to have perfect self-knowledge. This will cause them to fail on a lot of tasks where it intuitively seems like they should be able to succeed. Unless they can apply enough steam to fix the problem!
Conclusion
I have some slightly more formal math for steam worked out, but it doesn't capture everything above, so I thought it would be better to post the informal version for now.
I have some hope that this, or related concepts, will help solve some safety problems and/or make agency seem a bit less mysterious. I've gestured at connections to several important problems (procrastination paradox, spurious counterfactuals, troll bridge, goodhart, UDT, coordination problems). However, these connections are very speculative and I don't expect the whole picture to be convincing to readers yet.
I considered a lot of other terms besides "steam", but the English connotations of "steam" seem quite nice. Interesting to be borrowing ideas from the era where steam engines were the big new analogy for the mind and agency.