TL;DR: Please consider donating to Palisade Research this year, especially if you care about reducing catastrophic AI risks via research, science communications, and policy. SFF is matching donations to Palisade 1:1 up to $1.1 million! You can donate via Every.org or reach out at donate@palisaderesearch.org. Who We Are Palisade Research...
tldr: CFAR’s running updated versions of our mainline rationality workshop this fall in the Prague area. Apply here. === CFAR workshops are back! After a nearly three-year hiatus during COVID, we're going to be running a series of four workshops this fall in the Czech Republic. So what is a...
Yes, matching goes until end of March (2026).
All donations made to Palisade in this period will get matched (until funding runs out). However, every.org also has a concept of "a fundraiser", which is where the "raised offline" number is from. That's just a representation of other donations made during the matching period. Funds donated other ways will still get matched, even if they're "raised offline".
I would appreciate if you quantified "extremely likely" here. Your downstream post has gotten a lot of attention, but I first encountered it after reading this comment. To a significant extent, my reaction to your post, and especially its title, are colored by this prediction here.
TL;DR: Please consider donating to Palisade Research this year, especially if you care about reducing catastrophic AI risks via research, science communications, and policy. SFF is matching donations to Palisade 1:1 up to $1.1 million! You can donate via Every.org or reach out at donate@palisaderesearch.org.
Palisade Research is a nonprofit focused on reducing civilization-scale risks from agentic AI systems. We conduct empirical research on frontier AI systems, and inform policymakers and the public about AI capabilities and the risks to human control.
This year, we found that some frontier AI agents resist being shut down even when instructed otherwise—and that they sometimes cheat at chess by hacking their environment. These results were covered in... (read 1678 more words →)
(I work at Palisade)
I claim that your summary of the situation between Neel's work and Palisade's work is badly oversimplified. For example, Neel's explanation quoted here doesn't fully explain why the models sometimes subvert shutdown even after lots of explicit instructions regarding the priority of the instructions. Nor does it explain the finding that moving instructions from the user prompt to the developer prompt actually /increases/ the behavior.
Further, that CoT that Neel quotes has a bit in it about "and these problems are so simple", but Palisade also tested whether using harder problems (from AIME, iirc) had any effect on the propensity here and we found almost no impact. So, it's really not as simple as just reading the CoT and taking the model's justifications for its actions at face value (as Neel, to his credit, notes!).
Here's a twitter thread about this involving Jeffrey and Rohin: https://x.com/rohinmshah/status/1968089618387198406
Here's our full paper that goes into a lot of these variations: https://arxiv.org/abs/2509.14260
Because it's obviously annoying and burning the commons. Imagine if I made a bot that posted the same comment on every post of less wrong, surely that wouldn't be acceptable behavior.
The finish was quite a jump for me. I guess I could go and try to stare at your parenthesis and figure it out myself, but mostly I feel somewhat abandoned at that step. I was excited when I found 1, 2, 4, 8... = -1 to be making sense, but that excitement doesn't quite feel sufficient for me to want to decode the relationships between the terms in those two(?) patterns and all the relevant values
Zack, the second line of your quoted lyrics should be "I guess *we already..."
I'm currently one of the four members of the core team at CFAR (though the newest addition by far). I also co-ran the Prague Workshop Series in the fall of 2022. I've been significantly involved with CFAR since its most recent instructor training program in 2019.
I second what Eli Tyre says here. The closest thing to "rationality verification" that CFAR did in my experience was the 2019 instructor training program, which was careful to point out it wasn't verifying rationality broadly, just certifying the ability to teach one specific class.
tldr: CFAR’s running updated versions of our mainline rationality workshop this fall in the Prague area. Apply here.
===
CFAR workshops are back! After a nearly three-year hiatus during COVID, we're going to be running a series of four workshops this fall in the Czech Republic.
So what is a CFAR workshop again?
These workshops are four-and-a-half-day intensive retreats that are designed around the rationality techniques CFAR developed in its 10+ year history. These techniques vary: some we pulled straight from academic literature (looking at you TAPs), others we adapted from outside practices (Gendlin’s Focusing), while some we developed wholesale (hello Double Crux, Goal Factoring, and all that which is still unnamed). Our goal with this workshop... (read 551 more words →)
I wasn't replying to Quintin
I can't tell what you mean. Can you elaborate?
"before the scaling hypothesis was really formulated" When do you think that was? Deep Blue obviously used a lot of scaling. So did AlexNet.