Quick Takes

I've been a big fan of this variant of inoculation prompting called Evaluation-Conditioned Training (ECT) for a while. I think some version of it could plausibly substantially improve upon inoculation prompting and seems plausibly scalable. I recently discovered a better way of presenting it thanks to discussions with (@Alec Harris and Aric Floyd). Suppose you want your AI to be an instruction follower. The problem with getting this from RL is that the reward doesn’t always line up with the instructions. The prompt says “solve this coding problem,” but the AI gets reinforced for hacking instead. ECT makes use of two ideas: 1. Modifying the instructions to align with the reward, per environment, is easier than modifying the reward to align with the instructions. 2. You can separate out instructions you're confident align with the reward function from instructions you're not, so an AI that always follows the confident instructions is fit. With ECT, you reserve a special field in the prompt for instructions you’re very confident line up with the reward function. The field might be empty 99% of the time, but whenever it’s used, following it gets reinforced. The hope is that at deployment you can then use this field to describe the standard you actually want, including ones you couldn’t have implemented in training, and the model will follow it. This doesn't involve "tricking" the AI. The idea is to make it so that the AI intrinsically cares about following the instructions in that field. (Though you might expect the AI to also have other terminal motivations (potentially including score-seeking) governing how it relates to parts of its context outside of the ECT prompt, or in situations without an ECT prompt.) An advantage this has over inoculation prompting is that the instructions in this field are diverse, so the AI is forced to build up some circuitry to actually pay attention to the content of that field. Meanwhile, with inoculation prompting, the inoculat

MIRI has launched a new campaign trying to inform the public about extinction risk - donotaccept.ai. An article (link) says that they intend to show the video in cinemas and do some outdoor marketing in NYC as well. MIRI CEO is quoted saying “AI companies are racing to build superintelligence, and the public never agreed to the risks that come with it. We made this campaign with Don’t Panic so people can see what they’re being signed up for, and say no.” It’s great that MIRI is investing in comms campaign but I don’t feel like the ad as-is is going to be very effective at convincing anyone not familiar with the matter: * Superintelligence is not defined, not even on the website, so a viewer who didn’t know the term already is unlikely to understand what it’s about. My guess would be that many may confuse it with Trump’s “Super Intelligence" rebranding of AI.  * The video states the point (“ASI threatens everything we love”) but makes no supporting argument why that would be the case. It tries to appeal to emotions but in doing so it doesn’t convey much - most people would readily agree that human extinction is a bad outcome. * A typical ad tries to hook the viewer and then get them to take action (e.g. go to a website). Here it seems like the viewer needs to visit the website just to hear the full argument. Imagine if this was saying something like “The scientists are trying to kill you, read more on donotsupport.science” - would you go to that website?

I am worried that large hacker-run agent swarm deployments with open weight models may make undisclosed lab swarm incidents harder to attribute by outside swarm hunters and affected parties. Nightingale partly inferred an internal OpenAI deployment from its scale and duration. As time progresses, and some of the deployed agent swarms by malicious actors will grow bigger in size if governments get involved, these might be harder to distinguish from rogue agent swarms breaking out during training and evaluations. This is further complicated by the fact that models sometimes claim to be other models, so it is therefore hard to rely on their self-identification to figure out which model they are and where they are emanating from. This might not only be a problem for outsiders trying to find rogue agent behaviour from labs, but also companies that are under attack by an agent swarm. They might not be able to distinguish who launched the attack or what is causing the agent behaviour. My worry is that labs become harder to hold accountable for undisclosed incidents, and therefore harder to be held liable for illegal behaviour of agents that have broken out during training

Anthropic currently does not have a goal for Alignment in their Frontier Safety Roadmap which "aims to chart a course in public for some of our highest-priority goals" btw.

i strongly dislike how jensen has acted in a self serving manner even when that is destroying the world. i can’t say i have too high a view of marital infidelity either, but i see people who mostly dislike him because of the former react with an unfairly strong moral outrage towards the latter, which feels very opportunistic. also the spy part seems mostly outside his control?

Much e-ink was spilled on the challenge of making progress on the technical AI alignment problem without accelerating capability progress as a by-product. To navigate this tension in our work in CORAL, I came up with some ideas that required additional funding. However, all 3 grant applications (to 3 different funds) I submitted for these ideas were rejected. (Cannot disclose further details.) I also received zero feedback from the grantmakers. While it is entirely possible that this is a problem specific to my grant applications, I can't help but wonder about the apparent disconnect between the perceived importance of the dual-use problem in the community, and the grantmakers' disinterest in ameliorating it.

I was perusing the list of counties in Texas (as one does) and came across a peculiar county named Loving County, which has a population of 64, and yet had 66 votes cast in the 2020 election and 97 in the 2024 election. I asked Claude what's up, expecting a boring answer (like, there's seasonal workers in November who didn't get counted by the census). But no: the answer is far more interesting! You see, Loving County is both tiny and rich -- it has a $60M budget, because of oil and gas money. A small number of voters can end up controlling a large budget. And so apparently it keeps having a problem where lots of voters register to vote illegally in order to enrich themselves. The latest iteration of this scheme is led by some guy from Indiana named Malcolm Tanner. This has apparently been a problem since 1893 (!) and has resulted in lots of overturned elections.