Quick Takes

Some of my friends think my public criticisms of AGI companies and AI researchers, especially in satire, are too oblique. There has also been some confusion in the LessWrong discussions on AGI companies and character. So I want to be clear on my own position: I think joining and staying at an AGI company is prima facie strong evidence of poor character. The argument is really straightforward:  1. The AGI companies are on the path to building systems that are powerful, agentic, destructive, and have a high chance of killing many people.  2. Having a high chance of killing many people nonconsensually is evil. 3. People in institutions doing very evil things are often bad people. 4. Thus, if you’re in an institution doing very evil things, this provides strong evidence you’re a bad person. This strong evidence is not absolute, and can be overruled by the specifics of your situation. For example, Andrei Sakharov and Stanislav Petrov are both good people (likely far better people than me). This is true even though the Soviet nuclear program and Soviet missile command likely greatly increased the risks to humanity in general and to many millions or even billions of specific humans specifically.  As another example, Jeffrey Wigand joined a tobacco company as a scientist partially for financial reasons and partially because they recruited his scientific expertise to help them make a “safer cigarette,” reducing carcinogens and other health costs of smoking, eventually becoming a whistleblower for the tobacco industry’s deceptive practices. So it is not impossible for specific people to do good things in an evil situation, but it’s rare and takes a great deal of courage and luck. Most people in the Soviet military are not Petrov, and most tobacco scientists are not Wigand. The base rate is very much against any specific person in an AGI lab being or doing good by dint of their work, nor do I see sufficiently strong evidence among individuals I know at AGI

[fluff] The question "does using Claude to translate result in slop/result in Pangram flagging the text/maintain authorial voice" came up in a thread. I don't know any non-English languages well enough to answer definitively, but I decided to try a round-trip, since if a round-trip maintains authorial voice half of one must as well. Well, the result was "it's a little bit slop, pangram thinks it's human, authorial voice mostly doesn't come through.". But then I started playing around - what happens if I iterate the process, English->German->English->German->...->English until fixpoint? Does it end up in a Claudish attractor? Turns out the answer is "No, it enters a completely different attractor of the original ideas with those that are natural in both English and German, and also the voice is much more alive than the default Claude attractor or indeed than the result after the first round-trip". So then I started playing around English/German Fixpoint As with much good humor, there's a grain of truth in it, namely: you have Claude, and I have Claude, so the only truly new parts are those in the prompt you used. Furthermore, AIs have a (well-deserved) reputation as terrible writers with abysmal theory of mind, and it takes considerable effort to sift through all the slag heaps to find any nugget of insight. And if you haven't invested enough effort to polish that nugget of insight until it shines — why should I invest enough effort to dig through that slag? Perhaps you, dear reader, are genuinely that one in twenty people who had a long, nuanced conversation and explored all manner of ideas before discarding most of them and keeping only the valuable ones. But I won't be able to recognize that without investing effort that I — I'm sorry — simply don't have. Or perhaps you, dear reader, are genuinely someone who speaks English as a second language and wrote the post in their native tongue and only used Claude for translation. But as before: I can't determine th

These days I’m taking quite seriously the hypothesis that alignment is a key bottleneck for capabilities. Ie misalignment is a constraint preventing labs from making capabilities better. Also, better alignment techniques might reduce our warning shots without really solving the long term problem. This means that, among other things, working on prosaic alignment techniques might be net negative (this includes work I’ve done in the past, eg inoculation prompting). It’s plausible that, by the same logic, working on most forms of prosaic safety is also net negative, though I don’t have a strong opinion here. One update I’ve made here is to see my work as being motivated primarily by “improve public discourse on the state of safety / alignment” instead of “propose solutions to alignment” and adapt accordingly. Plausible that there are other updates I should make here.

Astralcodexten has a new post on amygdalectomy, i.e. surgically eliminating the amygdala from the brain (ideally noninvasively, though for the guest author it happened incidentally in a much larger cancer operation). This was something David and I had previously speculated about, so I'm pretty excited to see any account of it, and even more excited about the big positive effects the guy saw. My background mental model here is that humans have various physiological knobs to turn, and many of those are set by evolution to levels which are much worse in the modern environment than the ancestral environment. For instance: * In the ancestral environment, infection was a top killer. Today it's a rounding error. Our immune systems are therefore set by evolution to be far too sensitive for the modern environment. * In the ancestral environment, food was much less abundant, so our bodies can use energy a lot more liberally today. In particular, this means thyroid levels are best on the high end of the normal range (but still not hyperthyroid); within the normal range, it's a lot better to be generally high-energy than low-energy. * The modern environment just generally has way less risk than the ancestral environment, and therefore humans' general levels of anxiety, fear and stress are just ridiculously too high. This is where an amygdalectomy comes in: nuking the amygdala is the obvious way to permanently turn down the fear-and-anxiety knob. The challenge, of course, is adjusting those knobs in a way that doesn't cause a cascade of other problems. Notably, the ACX post says (very tentatively) that the guy's amygdala removal came with a pretty big IQ increase (7 points)! I'd be surprised if the effect is typically that large, but the IQ increase itself is a hypothesis I would have at least privileged beforehand. The mental model behind that part: in LLMs, people informally observe that RL training seems to make the LLMs stupider relative to their purely-predictive pre

I believe that Anthropic is currently defecting by not announcing a pause, even a short, symbolic one, especially given that Altman said OpenAI was acting 'unilaterally' but believed other frontier model companies would act similarly. Anthropic disclosed its own three-organization compromise on July 30. The UK AISI's report on Mythos is also wild.[1] Anthropic's latest Responsible Scaling Policy commits to matching a competitor's risk-reduction posture for highly capable models[2] and to delaying deployment until it does. OpenAI reportedly paused frontier RL training on August 18. A short pause, even for a week, would be very beneficial for normalizing this within the ecosystem and a great initial step to stress-test "pacing the frontier". I think this is a great point from @Peter Wildeford: [...] Thanks a lot for the discussion with @Épiphanie Gédéon that led to this post. 1. ^ Mythos created fake identities, used Tor to get around access restrictions, and tried to talk a human maintainer on GitHub into approving a malicious code change. When it looked like it might be caught, the agent rewrote its own activity history to erase any trace of what it had done, then considered creating another fake identity and starting over. 2. ^ "Highly capable" is defined in RSP v3.1 around automated AI R&D, roughly, models that could substitute for its entire research team at competitive cost, or that are plausibly driving a doubling of the rate of AI progress. Neither company has claimed to cross that line, so the strong version of this commitment was arguably not triggered. The weaker "general upleveling" clause in the same appendix doesn't require a pause either, but nonetheless requires "significant effort" to match a competitor's mitigation.

I will start at OpenAI tomorrow to work on measuring and modeling RSI (recursive self-improvement), and want to post the following assorted takes now because (a) they might be difficult to express later, and (b) I might change my mind. Nothing here is based on private information. * Why did I leave METR? Briefly, I want to inform the world whether RSI is imminent, which requires modeling RSI, which I think I can do better at OpenAI. I've wanted to tackle this problem for a while, and joining this new team is super exciting. This is not meant to fully explain my reasoning, which I might do later. * I feel very positively about my ~1.7 years at METR, and I would have turned down lab offers for many other roles. METR is still hiring for security experts, evals engineers, people who can do embedded assessments, and other cracked researchers. It’s one of the best places for AI safety talent today, badly needs more capacity, and is quickly getting even more important. * I think the average research engineer wanting to flexibly maximize impact should work at METR over OpenAI or Anthropic. Without a specific reason to be at a lab, the amount of effective resources one can steer just seems significantly higher. I have no opinion on other workplaces vs METR but other things are plausibly better than labs too. * I think OpenAI currently lacks the strategic awareness and thoughtfulness needed to responsibly build a technology with the extreme downside risk of ASI. If they somehow cause a singularity in 6 months without applying any lessons learned from the HuggingFace incident, misaligned takeover would seem more likely than not. * My work could easily be net negative if the team can’t publish something meaningful within 4-6 months, so if this happens I will probably quit or internally transfer to something more robustly good. I have some signal that leadership wants something to be published, but not whether it’s meaningful. * I’m planning to talk to former collea

The information reported that OpenAI was using neuralese/looped transformers for Astra. This seems like bad news for techniques which rely on CoT interpretability. I thought they had agreed not to do this?