Quick Takes

I don't feel confused by LLMs seeming very smart while being unable to automate hard work. Sometimes people find it mysterious or surprising that current AIs can't fully automate difficult tasks given how smart they seem. I don't find this very confusing. Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence. In humans, crystallized and fluid intelligence are very correlated due to limited time for learning and limited capacity for knowledge/memorization, but AIs can train for longer and (for unclear reasons) seem to have much higher capacity for knowledge/memorization. So, AI capabilities are often overestimated by naive comparisons. [1] I do find it somewhat mysterious/surprising that humans have such poor memory despite seemingly having so many parameters, and that some people have much better memory than others. My view is that this crystallized vs fluid distinction is maybe the first principal component of human vs AI differences and this explains a lot of the differences, but not all of them. We can add an additional "component" to our analysis of fluid intelligence by distinguishing between short-run fluid intelligence (we could call this "adaptation") and longer term fluid intelligence (this might effectively be sample efficiency of longer run learning). I think AIs are relatively stronger at short-term fluid intelligence than long-term fluid intelligence. To be clear, this doesn't mean I think extremely powerful AI isn't near: I expect full AI R&D automation [2] in <5.5 years [3] (and it seems 25% likely in <2.5 years). These AIs will likely retain an inhuman profile but will be trained heavily on relevant tasks (while still having much stronger fluid intelligence than current AIs). But pretty soon (0.25-4 years) after full AI R&D automation, I do expect AIs to exceed humans at e

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee. You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world. From my current perspective, this feels ominous in a kinda similar way as if I read a bunch of Marxist fiction from the 1800s and noticed that it always ended in a communist revolution. Spoilers for a range of ratfic (unfortunately telling you which ratfic itself constitutes a spoiler): EDIT: the spoiler tags seem not to be working on the images. Mods, can you help? For now I'll add a bunch of space below. Expect many spoilers if you scroll down. And some examples which (at least partially) subvert this trope:

Here are the 2025 AI safety papers and posts I like the most. The list is very biased by my taste, by my views, by the people who had time to argue that their work is important to me, and by the papers that were salient to me when I wrote this list. I am highlighting the parts of papers I like, which is also very subjective. (This is similar to the 2024 edition here.) Bangers on multiple dimensions ★★★ You can measure time horizon, and it grows predictably-ish (Measuring AI Ability to Complete Long Tasks) ★★★ Black-box techniques work better than you think + you can (and should) baseline interp techniques against them + you can check that an LLM “really” internalized misaligned properties via good generalization tests + many interp techniques don’t work that well (Auditing language models for hidden objectives, Eliciting Secret Knowledge from Language Models, Evaluating honesty and lie detection techniques on a diverse suite of dishonest models) ★★★ “Evil behavior” generalizes far + You can get far generalization beyond just “evil behavior”, and even from prod-like data + training against emergent misalignment can just make it more localized + inoculation prompting helps a lot against it (Emergent misalignment, Weird Generalization, Natural emergent misalignment from prod RL, Thought Crimes, Training a reward hacker despite perfect labels) Important ideas - the contribution I find most important is the idea ★★★ Secret loyalties and other forms of power concentration might be a big deal (How a Small Group Could Use AI to Seize Power, Gradual Disempowerment (arguments about power concentration are the ones I find the most convincing, but the essay makes stronger claims)) ★★ The best prioritized list of AI control threats (Prioritizing threats for AI control)  ★★ The future could come soon and would likely be dangerous (AI 2027) ★★ You could probably monitor compute enough to slow down AI development a lot (An International Agreement to Prevent the Premature Creat

Does anyone have takes on the new benchmark ARC-AGI-3? "Humans score 100%, AI <1% .... Most benchmarks test what models already know, ARC-AGI-3 tests how they learn"

 I received an automatic rejection for an article. I have been writing on LessWrong since the beginning. I did use AI to help write the article which is why the article was rejected, but I also spent considerable time working on it. I also used AI to help write an article I published in January, which now has 156 karma. https://www.lesswrong.com/posts/kLvhBSwjWD9wjejWn/precedents-for-the-unprecedented-historical-analogies-for-1 This new article is about why AI is going to destroy most jobs, and as I wrote in it, "This essay was written with help from AI. If I could not use AI productively to improve it, that would undermine either my argument or my claim to expertise." Finally, I had a stroke two years ago and have come to rely on AI when writing. Please allow me to publish this article. James Miller Professor of Economics, Smith College

Iran. As with Venezuela and Greenland, seems like much discourse is about very zero order things like 'he was/is a bad guy' and the rest about first order things like 'did/will US achieve objectives (whatever those are or get retconned to be) at low cost'. But surely in all cases the main effect is on the reputation of the US - as a state over the medium term, and of its current regime over the short? (E.g. perhaps a very compressed description might be moving from 'legalistic and acquisitive, with a cautious and moralistic streak' to 'capricious and vindictive, with a hot temper and a unilateralist bent'.) Whether that's good or bad is a whole different debate, and one I'm not seeing getting conducted! Any good sources? Is this more strategic than it first looks? To what extent are these naked principal-agent misalignments?

Small donors should not worldview-diversify. Occasionally I encounter small donors (e.g. 10% pledgers earning <$200K) with highly specialised skills and knowledge (e.g. working on a sub-sub-topic of an EA cause area) who donate primarily to GiveWell top charities. These people do incredible amounts of good, and are highly commendable. That said, I think would probably do more good by donating according to their inside view and special knowledge. Worldview diversification makes sense for large funders like Coefficient Giving, but their reasons don't apply to small donors: diminishing returns, cross-pollination of ideas, etc. If all the small donors switched to specialised donations, then the community overall would still be worldview diversified, where the diversification is happening across donors rather than within each donor. I think the optimal donation strategy for a small donor with domain expertise looks something like: 1. Save 10% in index funds by default. 2. Donate when you encounter an opportunity where (i) you can make a non-deferential case for funding it, and (ii) you are unusually positioned to evaluate or support it. This might happen once or twice a year. Paradigm cases: 1. Political donations 2. Opportunities that the institutional funders wouldn't fund, or it would harm them to fund 3. Areas where you think institutional funders lacks in-house expertise 4. Research that is illegible to generalist grantmakers but legible to you: * conceptual work requiring significant background to evaluate * a researcher you're convinced is excellent through direct interaction but who lacks credentials * a theory of change requiring many inferential steps 5. A project that needs bridging before it becomes legible for institutional funding 6. Volunteering, when there's some reason hiring you would be inconvenient Richard Ngo seems to follow something like this strategy (though he might qualify as a mid-sized donor). I disagree with some of