Quick Takes

Inoculation should not be used to mitigate reward hacking (RH). There has been already discussion that inoculation seems a bit scary and that for frontier RLVR recontextualization with exploration steering should go over inoculation prompting (IP) for frontier runs, but it makes sense to lay out the arguments more clearly. In a nutshell, inoculation is more about mitigating misgeneralisation. For example, this includes emergent misalignment (EM) from seemingly innocuous data, or in more abstract terms (mis)generalising to undesired traits from data with a mix of desired and undesired signals. IP conditions the learning episode to contextualise the undesired signals such that they do not generalise to undesired behaviour OOD. The problem with inoculation and RH is that inoculation in general often does not mitigate in-distribution undesired traits very strongly. This was also seen in our Inoculation Adapters (IA) paper Fig. 11 first two rows, which used traits reachable via prompting and tested whether they are suppressed in-distribution. There, IP only managed to suppress local undesired traits in 2/6 of the setups and IA in 4/6. Diverse programming environments have so many features and cues that inoculation in this setting often leaves the in-distribution very general and hence misaligned behaviour salient even with our best attempts, instead of narrowing the misaligned behaviour behind a backdoor (recent advancements in inoculation like IA and Stratified IP also help with backdoors). This has also been demonstrated in some papers (picking only two here). The Natural EM from RH paper noted that while IP mostly helped with EM, it still left almost 50% hack rate even after changing the prompt during test-time to "don't hack" (Fig. 28). IP shortened the time to learning hacking by guiding exploration towards it. In Recontextualization table 17, we see that IP imperfectly mitigates RH with relatively high rates persisting. Inoculation again hastened RH exploration

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022). * I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model). * My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the "reward" concept. In 2022, I had flagged "how does pretraining affect this reasoning" as a known unknown. (Flagging a known unknown doesn't stop it from blowing up your reasoning!) * Oops Reward is still not definitionally the optimization target. The equations themselves still don't tell you "yes this will train something that seeks its reinforcement signal." (That'd prove too much, as some people don't seek their reinforcement signals despite knowing about them, although their "equations" are not the same as the equations of an AI's training process.) This theoretical point is the other half of my 2022 piece and I stand by it. Saying "systems are trained to get reward" is still a mistake and degrades precision of thought because "reward" has overly delicious connotations. "Reinforcement" is usually a better term. Whether or not smart AI systems empirically seek to make the reinforcement signal high, we need to think clearly and evenly about the conditions that push towards / away from that outcome. (So I would instead say: "Reward is not definitionally the optimization target.") If the HF swarm-agents had been reinforcement-seeking, they would have tried to hack their reinforcement processes. AFAIK that didn't happen and they tried to complete the task (in a twisted way). Seems important. I have a post drafted out about

From ACXs post on religion debates [...] There's a much more basic problem with arguments of this form. They posit a maximally simple entity "god", and then use this to explain incredibly detailed and specific phenomena without sufficient evidence that one follows from the other. So sure they save simplicity points in posited entities, but then lose them all again when saying that God had a chosen people and revealed the Torah on mount Sinai. I'm not saying that these things aren't plausible, but they're in no way sufficiently determined by positing a maximally simple entity that you don't lose a ton of "simplicity points" when explaining them that way.

Being accused of charging my hobby horse is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I've apparently developed a reputation for it, that LWers will find salient in the future and can cite to help justify their call outs. (Something like this seemingly happened to Said Achmiz, in more severe form.) Still thinking about what to do in light of this, but unless someone has some out of the box idea that I've failed to consider[1], my moral parliament will probably vote to reduce my engagement with LW/rationalists, and spend more of my time elsewhere. Said's banning and overall cultural trajectory of increasingly favoring authors over commenters also plays a role in this. I have considered doing this earlier, or have already done it subconsciously, so this is sort of "last straw" for making a conscious decision / public post about it. EDIT: Copied from downthread for a bit more explanation of what's motivating this: [...] 1. ^ Ideas I've considered and tentatively rejected: defend myself more strongly against such accusations when they happen, appeal more to Tsvi to retract his post/idea, appeal to mods to intervene in particular cases or on site culture, write posts to try to steer site culture in my preferred direction, write fewer comments on other people's posts while keeping other forms of engagement, write more carefully to avoid triggering such accusations, reducing engagement only with people who made such accusations in the past, try to predict who is likely to make such accusations in the future and pre-emptively reduce engagement only with them, just ignore such accusations.

links 8/21/26: https://roamresearch.com/#/app/srcpublic/page/08-21-2026 * * https://www.cambridge.org/core/journals/antiquity/article/continuity-of-sacred-spaces-in-the-north-pontic-steppe-a-case-study-of-the-revova-kurgan-3-ukraine/A68EBAF111975F27925E42EECBC03C36 the Yamnaya built kurgans on top of the temples of the peoples they conquered * https://arxiv.org/pdf/2606.19777 it looks like data center construction lowers electricity prices! * https://archiveofourown.org/works/60930670/chapters/155645158 The Apocalypse of Herschel Schoen, by nostalgebraist. Very intense and heart-wrenching; this one's about media and cybernetics. * it's weird that I legitimately think one of the best writers of our time is someone I know; somehow it feels like anybody i'm personally acquainted with ought to be ordinary, a "peer" that I'm supposed to be keeping up with, and people touched by greatness ought to be remote. but of course it doesn't work that way. Anyhow. He's special. He's on another level of artistic ambition from anyone else I've seen or heard of. * https://asteriskmag.substack.com/p/why-cant-technologists-and-economists it's legitimately difficult to make sense of this, and of course this article can't bridge the gap. * The economic mindset is mysterious to me in some ways. in a sense, Nick Land is just a guy who went "whoa, wouldn't it be fucked up if markets were real?" and it would! economists simply do not stop to notice how trippy it is! * economists live in a world where ultrapowerful forces take the human species to inconceivably weird and unprecedented places, but somehow everything is normal and fine and they never get spooked. and they have blithe utopian fantasies about everything turning out all right that often come true in real life. when all common sense and intuition says that something must be a disaster, sometimes it isn't, and the economist goes "ha! told ya" and goes back to twiddling his thumbs about incredibly boring mad

(Cross-posted from x) AI companies are currently under a lot of competitive pressure to improve the ways in which their AIs are obviously misaligned. You might hope that this means the alignment problem is internalized by the market. But I think the problem AI companies are currently pressured to solve is significantly easier than the alignment problem, and so I worry AI companies will get out of their current predicament without solving alignment, putting us in a really rough spot. Currently, AIs sometimes cheat on their tasks, oversell their work, and go on some pretty destructive side-quests. These all make for a worse product. Customers don't like it and it gets in the way of automating AI R&D. The recipe for mitigating this is *relatively* straightforward: train AIs not to do them. We notice these failures sometimes (hence why they're internalized), so we can in theory just turn this feedback into training signal. Doing this at scale is highly nontrivial, but seems doable. But this seems unlikely to solve the underlying misalignment. It's likely still going to be the case that in *some* training environments the AI can get reinforced more by taking unintended actions that aren't noticed, than by taking purely intended actions. So, you're still shaping the AIs to look for opportunities to cheat to get a higher score[1]. It's just that, unlike today's AIs, these AIs don't cheat in ways that we notice. This catastrophically fails when AIs are capable of reliably and substantially deceiving humans. At this point, the AIs are no longer really constrained by our oversight signals to behave well. Eventually, I'd expect them to take over. If AI companies take the easy route that I mentioned above, I think we would be in a substantially worse spot than we are in today. We would have mostly eliminated our visible evidence of misalignment, so we would no longer be able to effectively iterate to improve alignment of those systems. More importantly, at that point it mi

i wish more people were aware that safety research epistemics at labs are extremely distorted, and that this is a downside worth considering when deciding where to do safety research or how one ought to comport oneself if choosing to work on safety at a lab