(Self-promotion warning.) Alexander Gietelink Oldenziel pointed me toward this post after hearing me describe my physics research and noticing some potential similarities, especially with the Redundant Information Hypothesis.  If you'll forgive me, I'd like to point to a few ideas in my field (many not associated with me!) that might be useful. Sorry in advance if these connections end up being too tenuous.

In short, I work on mathematically formalizing the intuitive idea of wavefunction branches, and a big part of my approach is based on finding variables that are special because they are redundantly recorded in many spatially disjoint systems.  The redundancy aspects are inspired by some of the work done by Wojciech Zurek (my advisor) and collaborators on quantum Darwinism. (Don't read too much into the name; it's all about redundancy, not mutation.) Although I personally have concentrated on using redundancy to identify quantum variables that behave classically without necessarily being of interest to cognitive systems, the importance of redundancy for intuitively establishing "objectivity" among intelligent beings is a big motivation for Zurek.

Building on work by Brandao et al., Xiao-Liang Qi & Dan Ranard made use of the idea of "quantum Markov blankets" in formalizing certain aspects of quantum Darwinism. I think these are playing a very similar role to the (classical) Markov blankets discussed above.

In the section "Definitions depend on choice of variables" of the current post, the authors argue that Wentworth's construction depends on a choice of variables, and that without a preferred choice it's not clear that the ideas are robust.  So it's maybe worth noting that a similar issue arises in the definition of wavefunction branches.  The approach several researchers (including me) have been taking is to ground the preferred variables in spatial locality, which is about as fundamental a constraint as you can get in physics.  More specifically, the idea is that the wavefunction branche decomposition should be invariant under arbitrary local operations ("unitaries") on each patch of space, but not invariant under operations that mix up different spatial regions.

Another basic physics idea that might be relevant is hydrodynamic variables and the relevant transport phenomena. Indeed, Wentworth brings up several special cases (e.g., temperature, center-of-mass momentum, pressure), and he correctly notes that their important role can be traced back to their local conservation (in time, not just under re-sampling).  However, while very-non-exhaustively browsing through his other posts on LW it seemed as if he didn't bring up what is often considered their most important practical feature: predictability.  Basically, the idea is this: Out of the set of all possible variables one might use to describe a system, most of them cannot be used on their own to reliably predict forward time evolution because they depend on the many other variables in a non-Markovian way. But hydro variables have closed equations of motion, which can be deterministic or stochastic but at the least are Markovian. Furthermore, the rest of the variables in the system (i.e., all the chaotic microscopic degrees of freedom) are usually "as random as possible" -- and therefore unnecessary to simulate -- in the sense that it's infeasible to distinguish them from being in equilibrium (subject, of course, to the constraints implied by the values of the conserved quantities). This formalism is very broad, extending well beyond fluid dynamics despite the name "hydro".

Further, assume that $X$ mediates between $\Lambda$ and ${\Lambda }^{\prime }$ (third diagram below).

I can't tell if X is supposed to be another variable, distinct from X_1 and X_2, or if it's suppose to be X=(X_1,X_2), or what. EDIT: From reading further it looks like X=(X_1,X_2). This should be clarified where the variables are first introduced. Just to make it clear that this is not obvious even just within the field of Bayes nets, I open up Pearl's "Causality" to page 17 and see "In Figure 1.2, X={X_2} and Y={X_3} are d-separated by Z={X_1}", i.e. X is not assumed to be a vector (X_1, X_2, ...).  And obviously there is more variability in other fields.

Other examples:

• “Career politician” is something of a slur. It seems widely accepted (though maybe you dispute?) that folks who specialize in politics certainly become better at winning politics (“more effective”) but that also this selects for politicians who are less honest or otherwise not well aligned with their constituents.

• Tech startups still led by their technical CEO are somehow better than those where they have been replaced with a “career CEO”. Obviously there are selection effects, but the career CEOs are generally believed to be more short-term- and power-focused.

People have tried to fix these problems by putting constraints on managers (either through norms/stigmas about “non-technical” managers or explicit requirements that managers must, e.g., have a PhD). And probably these have helped some (although they tend to get Goodhardted, e.g., people who get MDs in order to run medical companies without any desire to practice medicine). And certainly there are times when technical people are bad managers and do more damage than their knowledge can possibly make up for.

But like, this tension between technical knowledge and specializing in management (or grant evaluation) seems like the crux of the issue that must be addressed head-on in any theorizing about the problem.

Note that I'm specifically not referring to the elements of $A$ as "actions" or "outputs"; rather, the elements of $A$ are possible ways the agent can choose to be.

I don't know what distinction is being drawn here.  You probably need an example to illustrate.

Once you eliminate the requirement that the manager be a practicing scientist, the roles will become filled with people who like managing, and are good at politics, rather than doing science. I’m surprised this is controversial. There is a reason the chair of academic departments is almost always a rotating prof in the department, rather than a permanent administrator. (Note: “was once a professor” is not considered sufficient to prevent this. Rather, profs understand that serving as chair for a couple years before rotating back into research is an unpleasant but necessary duty.)

We see this with doctors too. As the US medical system consolidates, and private practices are squeezed to a tiny and tinier fraction of docs, slowly but surely all the docs become employees of hospitals and the people in charge are MBA-types. Some of them have MDs, and once practiced medicine, but they specialize in management and they don’t come back.

You can of course argue that the downside is worth the benefits. But the existence and size of the downside are pretty clear from history, and need to be addressed in such a system.

Letting people specialize as “science managers” sounds in practice like transferring the reins from scientists to MBAs, as was much maligned at Boeing. Similarly, having grants distributed by people who aren’t practicing scientists sounds like a great way to avoid professional financial retaliation and replace it with politicians setting the direction of funding.

UK’s proposal for a joint safety institute seems maybe more notable:

Sunak will use the second day of Britain's upcoming two-day AI summit to gather “like-minded countries” and executives from the leading AI companies to set out a roadmap for an AI Safety Institute, according to five people familiar with the government’s plans.

The body would assist governments in evaluating national security risks associated with frontier models, which are the most advanced forms of the technology.

The idea is that the institute could emerge from what is now the United Kingdom’s government’s Frontier AI Taskforce, which is currently in talks with major AI companies Anthropic, DeepMind and OpenAI to gain access to their models. An Anthropic spokesperson said the company is still working out the details of access, but that it is “in discussions about providing API access.”

https://www.politico.eu/article/uk-pitch-ai-safety-institute-rishi-sunak/

The softmax acts on the whole matrix

Isn't the softmax applied vector-wise, thereby breaking the apparent transpose symmetry? 

Strictly speaking, the plot could be 100% noise without error bars, sample size, or similar info. So maybe worth including that.

No. All the forms of leverage advocated in the book (e.g., call options and buying stocks on margin) at worst take your portfolio to zero if there is a huge market downturn. The book of course advocates keeping a safe rainy-day fund for basic expenses, like everyone else. So you don’t ever require a bailout. The idea is that having your retirement fund go to zero in your early twenties is hardly catastrophic, and the older you get the less leveraged you should be.