Written as a MATS extension fellow, mentored by Richard Ngo.[1]
Agents manage their beliefs differently to non-agentic predictors. A key example of agent-specific belief management is hyperstition, which is a particular case of goal-models.
Based on the idea that belief-management is an important aspect of agency, here's a (vibe) attempt at a rationality criterion:
An agent's beliefs are rational when they are a fixed point under the agent's own belief operator. B = U(B).
Here, U is an update (interpreted as the agent choosing to execute a belief update computation—or perhaps being forced to by a new observation).
This definition puts a huge load on the belief operator to be interesting. But I think it gives a clean conceptual separation between the instantaneous beliefs—which aren't that important—and the processes for updating them—which are.
This rationality criterion falls apart for embedded agents. Such an agent can only update in its own world, and that "act of doing inference" itself modifies the agent and its beliefs inside its world.
Consider an amended rationality criterion for embedded agents:
An agent's beliefs are (robustly) rational when they are a (robustly)near-fixed point under the agent's own belief operator. B = U(B) + epsilon(U,B)
Where epsilon is a controllable, bounded error term. When I say 'robustly,' I mean something like 'inside a stable attractor basin.' In practice, we probably won't have robustness; instead, an agent's beliefs will drift over time despite its best efforts—albeit in a locally gentle way (e.g. differentiably). This practical limitation becomes relevant later in the post.
The hallucinated superagent as an effective abstraction for rationality
Consider an agent that succeeds in bounding its own rate of drift by some gentle function. That agent would need to cultivate a harmonious relationship with its nearby past and future selves. This requires a great deal of "self-prediction," including:
reflecting on the intentions of your past self and using the affordances the past gave you to coordinate with it.
predicting the intentions of your future self and designing affordances for it to coordinate with you.
One technique for achieving this type of coordination is to follow the design of a Schelling-point-like superagent that you are confident your temporal variants will also respect.[2] The super-agent is nothing but a collective hallucination of the subagents: all of its computation is distributed across the temporal subagents, so any of its predictions about the lower layer are the same as predictions that the lower layers make about each other.
Scale-free hierarchical active inference
In case this cross-temporal arrangement fails to find a robust attractor basin, the hallucinated superagent is itself subject to drift. It can be seen as solving the same problem of reflective stability as the subagents, just on slower time-scales.
I felt rather mortified, at this point in my reasoning, to realise that I'm just reinventing hierarchical active inference from a bottom-up perspective. Data that flows upwards in the hierarchy are belief updates caused either by new observations or by lower levels conducting computations. Data flowing down the hierarchy are predictions the upper layer makes about the lower layer—which again are nothing more than a hallucinated summary of the predictions that members of the lower layer make about each other.
I motivated this picture in the case that subagents are indexed by time, but it generalises fairly painlessly to the usual multiagent settings. Formalising or developing the picture could substantiate hierarchical active inference (or a proximate cousin) as the emergent superagent produced by subagents recursively predicting each other.
Meta: I wrote this as a challenge to share more rough drafts and unfinished ideas. I endorse superficial aspects of this post less than I usually would for a Lesswrong main post (e.g. wording, stylistic choices, academic completeness, etc...). I decided to publish as a main post anyway because I am enthusiastic about the central idea.
I'm a little confused by the terminology here, if the pattern of agency across time is called a "superagent" then an "agent" only ever exists at one point in time, so it's really more of a "decision" than an agent?
Written as a MATS extension fellow, mentored by Richard Ngo.[1]
Agents manage their beliefs differently to non-agentic predictors. A key example of agent-specific belief management is hyperstition, which is a particular case of goal-models.
Based on the idea that belief-management is an important aspect of agency, here's a (vibe) attempt at a rationality criterion:
Here, U is an update (interpreted as the agent choosing to execute a belief update computation—or perhaps being forced to by a new observation).
This definition puts a huge load on the belief operator to be interesting. But I think it gives a clean conceptual separation between the instantaneous beliefs—which aren't that important—and the processes for updating them—which are.
This rationality criterion falls apart for embedded agents. Such an agent can only update in its own world, and that "act of doing inference" itself modifies the agent and its beliefs inside its world.
Consider an amended rationality criterion for embedded agents:
Where epsilon is a controllable, bounded error term. When I say 'robustly,' I mean something like 'inside a stable attractor basin.' In practice, we probably won't have robustness; instead, an agent's beliefs will drift over time despite its best efforts—albeit in a locally gentle way (e.g. differentiably). This practical limitation becomes relevant later in the post.
The hallucinated superagent as an effective abstraction for rationality
Consider an agent that succeeds in bounding its own rate of drift by some gentle function. That agent would need to cultivate a harmonious relationship with its nearby past and future selves. This requires a great deal of "self-prediction," including:
One technique for achieving this type of coordination is to follow the design of a Schelling-point-like superagent that you are confident your temporal variants will also respect.[2] The super-agent is nothing but a collective hallucination of the subagents: all of its computation is distributed across the temporal subagents, so any of its predictions about the lower layer are the same as predictions that the lower layers make about each other.
Scale-free hierarchical active inference
In case this cross-temporal arrangement fails to find a robust attractor basin, the hallucinated superagent is itself subject to drift. It can be seen as solving the same problem of reflective stability as the subagents, just on slower time-scales.
I felt rather mortified, at this point in my reasoning, to realise that I'm just reinventing hierarchical active inference from a bottom-up perspective. Data that flows upwards in the hierarchy are belief updates caused either by new observations or by lower levels conducting computations. Data flowing down the hierarchy are predictions the upper layer makes about the lower layer—which again are nothing more than a hallucinated summary of the predictions that members of the lower layer make about each other.
I motivated this picture in the case that subagents are indexed by time, but it generalises fairly painlessly to the usual multiagent settings. Formalising or developing the picture could substantiate hierarchical active inference (or a proximate cousin) as the emergent superagent produced by subagents recursively predicting each other.
Meta: I wrote this as a challenge to share more rough drafts and unfinished ideas. I endorse superficial aspects of this post less than I usually would for a Lesswrong main post (e.g. wording, stylistic choices, academic completeness, etc...). I decided to publish as a main post anyway because I am enthusiastic about the central idea.
FDT