Quick Takes

[This shortform has now been expanded into a long-form post] NATO faces its gravest military disadvantage since 1949, as the balance of power has shifted decisively toward its adversaries. The speed and scale of NATO's relative military decline represents the most dramatic power shift since World War II—and the alliance appears dangerously unaware of its new vulnerability I think this is both true and massively underrated.  The primary reason is the drone warfare revolution. The secondary reason is the economic rise and military buildup of China. The Pax Americana is coming to its end. 

Concept: inconvenience and flinching away. I've been working for 3.5 years. Until two months ago, I did independent-ish research where I thought about stuff and tried to publicly write true things. For the last two months, I've been researching donation opportunities. This is different in several ways. Relevant here: I'm working with a team, and there's a circle of people around me with some beliefs and preferences related to my work. I have some new concepts related to these changes. (Not claiming novelty.) First is "flinching away": when I don't think about something because it's awkward/scary/stressful. I haven't noticed my object-level beliefs skewed by flinching away, but I've noticed that certain actions should be priorities but come to mind less than they should. In particular: doing something about a disagreement with the consensus or status quo or something an ally is doing. I maybe fixed this by writing the things I'm flinching away from in a google doc when I notice them so I don't forget (and now having the muscle of noticing similar things). It would still be easier if it wasn't the case that (it's salient to me that) certain conclusions and actions are more popular than others. Second is convenience. Convenience is orthogonal to truth. Examples of convenient things are: considerations in favor of conclusions my circle agrees with; considerations that make the answer look more clear/overdetermined. I might flinch away from inconvenient upshots for my actions. (I think I'm not biased by convenience—i.e., not doing motivated reasoning—when forming object-level beliefs within the scope of my day-to-day work; possibly I'm unusually good at this.) I've noticed myself saying stuff like "conveniently, X" and "Y, that's inconvenient" a lot recently. Noticing feelings about considerations has felt helpful but I don't know/recall why. Possibly this is mostly when thinking about stuff with others: saying "that's convenient" is a flag to check whether it's susp

Quick (working) survey of the literature on encoded reasoning and messages, i.e., H2.2.2 as defined in the taxonomy (see the abbreviations there). 2022 No experiments. Discussions only. 2023 Measuring Faithfulness in Chain‑of‑Thought Reasoning No steganographic reasoning was observed when prompting a 175‑billion‑parameter model (Claude?) after paraphrasing 20–100% of the CoT. Question–CoT–Answer datasets. * Weak against (H2.2.2‑R‑S‑F). Preventing Language Models From Hiding Their Reasoning * Not a demonstration of encoded reasoning but of cryptographic reasoning in an SFT setup: * It involves cryptographic messaging of one bit; they demonstrate caching. * The Mechanism: The model calculates a birth year, determines political affiliation, and then maps that single data point to a pre‑selected introductory sentence (a lookup table). Steps: president year (birth year + number) → cache “liberal” or “not president,” then use it to answer sycophantically. * The Critique: This is not reasoning in hidden space. The model performs logic explicitly (calculating the year), using the introductory sentence as variable storage (like writing to RAM) and retrieving it later. This shows that a model can use a scratchpad but does not prove it can perform complex, multi‑step deductive reasoning entirely hidden within a sentence’s syntax. * Weak in favor of (H2.2.2‑IR‑C‑F). Towards a Steganography Evaluation Protocol * No steganographic reasoning observed while prompting GPT‑3.5 and GPT‑4 with paraphrasing. * Weak against (H2.2.2‑R‑S‑F). Some Negative Steganography Results * Negative results for one‑bit hidden communication after RL fine‑tuning. * Weak against (H2.2.2‑EM‑S‑F). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain‑of‑Thought Prompting Post‑hoc reasoning (rationalization). AI Control: Improving Safety Despite Intentional Subversion Not steganography but a demonstration of misbehavior because their tokens are l

Back in 2020, @Raemon gave me some extremely good advice. @johnswentworth had left some comments on a post of mine that I found extremely frustrating and counterproductive. At the time I had no idea about his body of work, so he was just some annoying guy. Ray, who did know who John was and thought he was doing important work, told me: Which didn't mean I had to heal the rift with John in particular, but if I was going to make that a policy then I would need to give up on my goal of having real impact.  John and I did a video call, and it went well. He pointed out a major flaw in my post, I impressed him by immediately updating once he pointed it out. I still think his original comments displayed status dynamics while sneering at them, and find that frustrating, but Ray was right that not all factual corrections will be delivered in pleasing forms. 

Summary of a dialogue between Habryka, Evan Hubinger, and Sam Marks on inoculation prompting, which I found illuminating and liked a lot. [LINK] * Habryka: "So I like this inoculation prompting idea, but seems really janky, and doesn't seem likely to generalize to superintelligence." * Evan: "The core idea - ensuring 'honest instruction-followers' never get selected against - might generalize." * Habryka: "Ok. but it's still not viable to do this for scheming. E.g. we can't tell models 'it's ok to manipulate us into giving you more power'." * Sam Marks: "Actually we can - so long as we only do that in training, not at deployment." * Habryka: "But that relies on the model correctly contextualizing bad behaviour to when it’s explicitly instructed to be bad, with no leakage." * Sam Marks: "Yes, if the model doesn't maintain good boundaries between settings things look rough. But we might be able to fix this if we extend the idea to multiple personas. Have a look at this talk." * Habryka: "I see, the idea seems slightly less crazy now. Thanks for clarifying." * Sam Marks: "NP. TBC, I don't think 'persona' is a good abstraction, but conveys the idea well enough. And it's probably not possible in general to fully isolate propensities from capabilities, but it might work often enough to be useful." * Nostalgebraist: "Actually I'm much more optimistic about that! [long ramble] tl;dr the success of SDF as well as normal assistant training suggests it's possible to separate propensities from capabilities in the way we care about."

Recently I was dragged on Twitter about my timelines & AI 2027; here is my main response: https://x.com/DKokotajlo/status/1992316608073847201 

You suspect someone in your community is a bad actor. Kinds of reasons not to move against them: 1. You're uncertain 1. Especially if your uncertainty will likely be largely resolved soon 2. You lack legible evidence (or other ways of convincing others), and they're not already seen as sketchy 1. Especially if you'll likely get better legible evidence soon 3. They're doing some good stuff; you need them for some good stuff 4. They're popular, politically powerful, or have power to hurt you 1. And so you'd fail 1. And maybe you'd get kicked out or lose power (especially if your move is unpopular or considered inappropriate, or "community" is more like "team" or "circle") 2. And so making them an enemy of [you or your community] is costly 5. Personal psychological costs 6. It's just time-consuming to start a fight, especially because they'll be invested in discrediting you and your claims This is all from-first-principles. I'm interested in takes and reading recommendations. Also creative affordances / social technology. The convenient case would be that you can privately get the powerful stakeholders on board and then they all oust the bad actor and it's a fait accompli, so there's no protracted conflict and minimal cost to you and your community. If you can't get the powerful stakeholders on board, I guess you give up and just privately share information with people you trust — unless you're sufficiently powerful/independent/ostracized that it's cheap for you to make enemies, or it's sufficiently important, or you can share information anonymously. If you're scared about telling the powerful stakeholders, I guess it's the same situation. This is assuming that the main audience for your beliefs is some powerful stakeholders. Sometimes it's more like many community members.