Quick Takes

Some beliefs I have about AI superpersuasion: * Persuading people at a level somewhat above top humans will be possible by end of 2027 at current rates of progress. * This will probably require audiovisual channels, because the output space is much larger than text. * We are already seeing catchy animations from Claude 5.5. In most domains, when models are kind of good at something, they saturate it within a year. * Persuading people of literally anything whatsoever will be possible when models are wildly superintelligent (250+ ECI). * Persuading people of things in their interest (i.e. that an uninfluenced but well-informed copy of them would endorse) is much easier than persuading people to do things that harm them. * Humans will vary in their vulnerability to superpersuasion. * People who are more informed (e.g. LW readers vs general public) will be less vulnerable. * People who spend more time spend socializing with AIs will be more vulnerable, just like with AI psychosis. * For a while, people will learn to ignore each generation of superpersuasive AI content the way they ignore banner ads. * However, the history of propaganda suggests that superpersuasive content will still have an effect even if people know it's AI-generated. * At some capability level, superpersuasive power will exceed anyone's ability to ignore it. It will just be too engaging, first like TikTok or Twitter, then like heroin. * Remember the average American already spends 6h45m/day watching TV and digital video, up from 5 hours in 2010. * Superpersuasion is a serious loss of control risk due to social engineering. * A superpersuasive, misaligned model could gain power and human allies at an AI lab even if it can only make positive-sum deals with employees. * We can reduce this risk in the near term through security protocols similar to those used to prevent social engineering or phishing. * We are fortunate that models aren't superhuman at persuasion yet, g

Belated thoughts on AB 316: how the heck did I miss this? When reading LASST's new lawsuit against OpenAI, I learned of Cal. Civ. Code § 1714.46(b), a recent addition to California civil liability law that says: [...] This was added in California Assembly Bill 316, which was signed into law on October 13, 2025, 2 weeks after SB 53 was signed into law. I'm surprised that I missed AB 316 -- I don't remember anyone talking about it, and I couldn't find mentions of it in a quick search on either LessWrong or Zvi's substack. Of course, this is civil liability law, and really all it does is bar one specific defense. But as the LASST lawsuit shows, this rule might be very important for how the courts handle autonomous AI incidents in the future. Whoops. So, how did I miss this? My brief postmortem: 1. The bill AB 316 was framed as a consumer safety/child protection bill. For example, it was sponsored by the Children’s Advocacy Institute and the Organization for Social Media Safety, and the Assemblywoman who introduced this mentions only child self-harm in her statement on the bill's signing. It was passed around the same time that many other pieces of AI legislation, including SB 53, were passed. So I didn't notice it when it passed. 2. It's not criminal law like CDAFA or a regulatory statute like SB 53 or the UCL. When researching the legal questions relating to the Hugging Face incident, I didn't think to check civil liability law. In response to this, I've set up a Claude instance to periodically read through and summarize all the AI-related bills that are passed. I've also skimmed through all the pieces of California legislation listed in the AI Law Tracker. (I didn't see any other relevant-to-frontier-AI bills that I wasn't aware of. Not sure whether to be thankful or sad about that.)

Why is (Updateless) Decision Theory so popular around here? The LessWrong sphere has long had a fascination with decision theory. Despite hanging out around there parts for many years I still don't understand Decision theory. I must also admit that despite being a big theory-stan I don't know if I could make a convincing case that decision theory is important (for AI alignment). Nor do I really understand why updateless decision theory (or its various cousins like logical, functional, udt 1.0, 2.0 etc I must admit I don't understand the difference between these ) in particular is so relevant. I asked Yudkowsky directly and after some prodding he agreed there were three motivations: 1. Basic Science of Agent Foundations 2. AIs will collude against you even when you set them up against each other. Incentivizing different AIs to compete against each other won't work. Many people when first hearing about the alignment problem come up with various safety schemes that comes down to incentizing different AIs to compete with the AI controlling the mechanism. This is problematic: it is more natural for very powerful agents to band together against the weaker agents. The usual solution that is proposed is to come up with a clever reward scheme that directly incentives AIs to compete, eg by ratting each other out to the humans. This is still problematic because very capable minds can coordinate even in prisoner's dilemma [Open-Source Game theory] 3. Reflective stability - alignment is not just about aligning the current AI. The current AI will build a new AI or self-modify upon reflection. In other words, you need to find a good reflectively stable fixed point. Decision theories that are not Yudkowsky-approved are not reflectively stable. There is a fourth reason I've seen 4. Anthropics - (updateless) decision theory shows up when you start having Big thoughts about the universe and our place within it. Big If True. I like these reasons. I also can't shake the feeling

I was requested to submit a statement to today's US Senate hearing on "Rogue AI: Securing the Homeland Against AI Agent Attacks." Watch at 2:30PM Eastern. Here's my statement: [...] 1. ^ https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyber-capable-agents/ 2. ^ https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-australian-government-site-pm-says/107189078 3. ^ https://turntrout.com/why-i-left-google-deepmind#does-this-have-any-impact-on-existential-risk-from-ai

First pass research by Claude on the "all frontier AI labs unionize and the slowdown happens that way" proposal is that union contracts that make companies make business decisions (such as outside contracts or model releases) aren't allowed, but unions can push for more published transparency, whistleblower protections, and the right to refuse to participate in doing things that would cause legal liability (such as unsafe unsandboxed test runs). I might be able to get a second pass opinion from someone with union law experience, but no promises.

I should probably mention somewhere publicly that I don't think unaligned AI takeover necessarily means human extinction[1]. I don't usually fight about it because a) I think unaligned takeover that doesn't lead to human extinction is still very bad and means ~all value from the affectable universe is gone b) strategically I think it's unwise to fight people more pessimistic than me on this since there are very few downstream differences in prioritization, c) it's slightly ghoulish and unnecessary to think about. I want to be clearer about this now because I'm trying out talking about my views more with more general audiences. And if you see me talk about takeover and related problems without foregrounding extinction, you should assume it's representing my actual beliefs (if emphasized differently from how I'd emphasize it on LessWrong), rather than me downplaying extinction threats for optics reasons. I also want to write this clearly such that if I do change my mind in the future it'd clearly be an actual mind change rather than something I'll wiggle out of later, albeit from one position that I don't think is very important to another that I don't think is very important. 1. ^ My current best guess for literal extinction conditional upon takeover is somewhat but not dramatically under 50%. I mean extinction in a very commonsensical way, like would at least 150-500 biological humans survive 30 years after takeover or something.

A felt experience of gradual disempowerment: some humans are becoming promptable (in an explicit way distinct from “influence”, ”interrogation” etc.) I’m encountering people in my work who are now heavily using AI models to filter all of their thoughts, our work context, and send replies - I.e most of their responses are 100% AI generated. If I state opinionated takes assertively, it is incorporated in their work as fact without pushback or discussion! Interested whether anyone has experienced similar