It should! Fixed, thank you :)

The UK Government tends to use the PHIA probability yardstick in most of its communications.

This is used very consistently in national security publications. It's also commonly used by other UK Government departments as people frequently move between departments in the civil service, and documents often get reviewed for clearance by national security bodies before public release.

It is less granular than the IPCC terms at the extremes, but the ranges don't overlap. I don't know which is actually better to use in AI safety communications, but I think being clear if you are using either in your writing seems a good way to go! In any case being aware it's a thing you'll see in UK Government documents might be useful.

A comment provided to me by a reader, highlighting 3rd party liability and insurance as interventions too (lightly edited):

Hi! I liked your AI regulator’s toolbox post – very useful to have a comprehensive list like this! I'm not sure exactly what heading it should go under, but I suggest considering adding proposals to greatly increase 3rd party liability (and or require carrying insurance). A nice intro is here:
https://www.lawfaremedia.org/article/tort-law-and-frontier-ai-governance

Some are explicitly proposing strict liability for catastrophic risks. Gabe Weil has proposal, summarized here: https://www.lesswrong.com/posts/5e7TrmH7mBwqpZ6ek/tort-law-can-play-an-important-role-in-mitigating-ai-risk

There are also workshop papers on insurance here:
https://www.genlaw.org/2024-icml-papers#liability-and-insurance-for-catastrophic-losses-the-nuclear-power-precedent-and-lessons-for-ai
https://www.genlaw.org/2024-icml-papers#insuring-uninsurable-risks-from-ai-government-as-insurer-of-last-resort

NB: when implemented correctly (i.e. when premiums are accurately risk-priced), insurance premiums are mechanically similar to Pigouvian taxes, internalizing negative externalities. So maybe this goes under the "Other taxes" heading? But that also seems odd. Like taxes, these are certainly incentive alignment strategies (rather than command and control) – maybe that's a better heading? Just spitballing :)

I don't understand how the experimental setup provides evidence for self-other overlap working.

The reward structure for the blue agent doesn't seem to provide a non-deceptive reason to interact with the red agent. The described "non-deceptive" behaviour (going straight to the goal) doesn't seem to demonstrate awareness of or response to the red agent.

Additionally, my understanding of the training setup is that it tries to make the blue agent's activations the same regardless of whether it observes the red agent or not. This would mean there's effectively no difference when seeing the red agent, i.e. no awareness of it. (This is where I'm most uncertain - I may have misunderstood this! Is it to do with only training the subset of cases where the blue agent doesn't originally observe the red agent? Or having the KL penalty?). So we seem to be training the blue agent to ignore the red agent.

I think what might answer my confusion is "Are we just training the blue agent to ignore the red agent entirely?"

• If yes, how does this show self-other overlap working?
• If no, how does this look different to training the blue agent to ignore the red agent?

Alternatively, I think I'd be more convinced with an experiment showing a task where the blue agent still obviously needs to react to the red agent. One idea could be to add a non-goal in the way that applies a penalty to both agents if either agent goes through it, that can only be seen by the red agent (which knows it is this specific non-goal). Then the blue agent would have to sense when the red agent is hesitant about going somewhere and try to go around the obstacle (I haven't thought very hard about this, so this might also have problems).

Most sunscreen feels horrible and slimy (especially in the US where the FDA has not yet approved the superior formulas available in Europe and Asia).

What superior formulas available in Europe would you recommend?

Thanks for the feedback! The article does include some bits on this, but I don't think LessWrong supports toggle block formatting.

I think individuals probably won't be able to train models themselves that pose advanced misalignment threats before large companies do. In particular, I think we disagree about how likely we think it is that there's some big algorithmic efficiency trick someone will discover that enables people to leap forward on this (I don't think this will happen, I think you think this will).

But I do think the catastrophic misuse angle seems fairly plausible - particularly from fine-tuning. I also think an 'incompetent takeover'^[1] might be plausible for an individual to trigger. Both of these are probably not well addressed by compute governance (except maybe by stopping large companies releasing the weights of the models for fine-tuning by individuals).

1. ^{^}

   I plan to write more up on this: I think it's generally underrated as a concept.