Quick Takes

It's incredibly easy to be fooled by the capabilities of the current top-performing tech (LLM agents). It's easy because they have a vast amount of training data to interpolate from. This works fine to acquire capabilities within our existing data distribution of the world (one that is also easy to verify), but what happens when they go out of distribution? LLMs perform poorly! Yet, people seem to think they can actually generalize to new problems. Why is that? It's, again, the vastness of their training data. It makes it hard to distinguish between interpolation and extrapolation (or hyperpolation, if you want to add a third dimension). For example, a Typescript app is within-distribution! AI research in the existing body of research is within-distribution, and companies are paying millions to build RL environments to make them *specifically* good at some of those things! Related and great post from Beren, "Most Algorithmic Progress is Data Progress": [...] It might still be impressive, but models are largely remixing many things it has seen in great detail during training (many impressive headline results have even been determined to be the model re-using existing implementations/PRs via search instead of coming up with actually-new ones!). This is not about LLMs not doing impressive things! This is about precisely describing their capability profile, where it comes from, and whether more of the same (e.g., scale) gets you a whole new set of impressive outcomes (e.g., novel R&D that isn't just remixing existing research). And yes, I know you can make a ton of discoveries by interpolating existing research (e.g., interdisciplinary research and automating research pipelines to run more experiments). I also think that people are overly confident that it means LLMs will be capable of novel R&D breakthroughs, and what that means is needed from future AIs. Even if you consider "researchers can come up with novel ideas and give them to the AIs", that likely invo

a theory of assistant personas and superhuman capabilities so you have a language model. you train it to embody some specific personality--Claude, ChatGPT, whatever. one of the miracles of AI is that this mostly works and gives you something that is mostly trying to help you and not trying to murder you. i claim that this is mostly because of the SL training objective and if you do just the intense RL thing you get the originally predicted spicy alignment failures. suppose you tell the LM that Claude is actually a superhuman aligned AI. can you get superhuman capabilities from Claude? an obvious upper bound is the capabilities of the language model, so it begs the question of how those superhuman capabilities got in the model in the first place. maybe in the limit of compute your language model will understand everything and know how to do everything, but in practice everyone agrees this would be a horribly inefficient way to get truly superhuman capabilities. rather, in practice people take LMs and also do a bunch of RL on verifiable domains. what happens then if you start with a model role playing an aligned assistant but then try to train it to have superhuman capabilities? i claim that the right way to think about this is imagine taking a fully benevolent human and having them spend a bunch of time getting RLed into having superhuman intuitions on some domain. for example, maybe you put them in the Business Simulator and they learn to build extremely successful companies. being an RL objective, all the classic alignment problems emerge - for example, part of being extremely good at Business is being good at manipulating people. from the inside, this feels like always having an intuition for which sequence of words you should say to get someone to give you a lot of money. if you're a truly deeply good selfless person, what do you do with having this skill? you could just ignore it. but that's leaving a lot on the table. maybe you can listen to it very very car

How to turn on uploading your photos to Google cloud: * at a random moment (maybe a fraction of a second before you clicked on your photo) a dialog box appears asking you whether you want to upload your photos to Google cloud: the options are "yes" and "keep nagging me in random intervals later", clicking outside of the dialog also counts as "yes" (so maybe you have 'consented' without even having an opportunity to read what the text says) How to turn off uploading your photos to Google cloud: * open Google Photos app on your smartphone * click on your profile photo * click "manage Google backups" * change the switch to "do not backup" * open Google Photos web page in your browser * select all photos * click "delete" * go to the recycle bin * click "delete" I am happy that I have an AI to figure this out. When you already know how to do it, it seems obvious in hindsight, but if you previously didn't even know that there was such a thing as a Google Photos web page (because you only knew the smartphone app), it was much less obvious.

Cross-posting from a Twitter thread responding to a recent viral comments by @Richard_Ngo about EA, Anthropic, and AI safety as a 'fake field.' Posting here because I expect this to be quite unpopular on LW. (original thread: https://x.com/CRSegerie/status/2056737155880493357) AI safety in 2023–2026 was driven by evals, threat models, scary demos, model-organism work, RSPs, and voluntary commitments. Richard calls this "much more of a fake field" and says it "won't generalize". Here's why I disagree - 1/10 1/ I agree with Anthropic being now the biggest lever. They lead the AGI race, and Mythos moved the White House; this is quite a feat! But many of the specifics are wildly overstated 2/ Not a blind spot. Empowering safety-conscious actors at the frontier was openly debated on the forum for years. Calling a deliberate/contested strategy a "blind spot" rewrites history. The bet was visible and explicit. Personally, I've publicly criticized Anthropic on a few topics, but I still think the field is in a much better position, given that they're leading compared to the shady behavior at OpenAI. 3 /The effect of Anthropic leading is not just "AGI faster" Anthropic has many positive externalities: * Dario has been more candid than most CEOs about risks in public (even if he could still go a lot further) * They are doing top-tier research and implementing SOTA mitigations I don't know what I would have done with Mythos at their place. In the past, when I've discussed this with people at Anthropic, I've often updated on the difficulty of being in the driver's seat. I might be wrong, but I don't think it would be easy to improve Anthropic's behavior qualitatively in a game-changing way (even if many substantial improvements are on the table). 4/ Anthropic visibly moved US executive posture, Senate hearings, frontier-lab norms, and the public conversation toward taking the risks seriously. Yes, they relinquished their RSPv2, and we no longer have the guarant

"to the success of our hopeless cause" is such a good toast and we should use it more often. i first learned of it from the book of the same name, and apparently it was a common refrain at gatherings of Soviet dissidents. i like it because it captures the feeling of trying really hard to succeed despite being in the basement of the logistic success curve, and somehow, despite all odds, actually succeeding in the end.

the following fictional dialogue is a complete unapologetic strawman but it's funny enough i had to bring it into being: ---------------------------------------- “So I asked myself: where can I make the most impact? And clearly malaria is the most important area.” “And so you decided to donate all of your money to buy malaria nets?” “Well, so it turns out that saving lives from malaria is actually kind of expensive and indirect. You see, it costs thousands of dollars to save a life. Statistically. Who knows if you’re actually changing anyone's life that way?” “And so you found a more efficient way to save lives.” “Actually, it turns out that it’s cheaper to give people malaria. It's a lot more impactful and the technical problems are more interesting.” "I see. Isn't more malaria bad though?" "I don't know, but I find it much easier to work on because the feedback loops are much tighter. Maybe one day, if malaria gets big enough, I’ll go work on saving people from malaria. But we're still a long way away from everyone having malaria."

I've been reading an anthropology about the Australian aboriginals, The Native Tribes of Central Australia (Baldwin Spencer/Francis James Gillen, 1899), and found some parts interesting enough to share them. Content warning: description of gruesome (though consensual) mutilation. Things that stood out about the aboriginals (highlighting not in the original text): * Aboriginals experienced nocebo effects strong enough to result in death, even from mild injuries, if the weapon causing the injury was believed to be enchanted [1] * Each aboriginal man had the right to at least one wife [2] * Aboriginals were very good at tracking, for example easily able to distinguish individuals from their tracks [3] * The central Australian aboriginals plausibly had a form of specialization/division of labour independent of differences in supply of resources [4] * Aboriginals had a lot of rituals resulting in injuries, sometimes gruesome ones, the cost of not engaging in these rituals in ridicule, which is highly aversive [5] * The most shocking one was penile subincision (extra content warning, very unpleasant images of mutilated penises) * Basically, penile subincision is a cutting-open of the urethra along the length of the penis starting from the tip, differing in how far it is cut [6] * Very surprisingly to me, many men who willingly undergo the subincision a second (and even third!) time [7] * In order to become capable of magic, aboriginals would make a hole in their tongue [8] (without, apparently, any guidance on how to do that) and push small stones far under their fingernails [9] * One minor ceremony involved the knocking out of one or more teeth, both in men [10] and women [11] * Commentary: I find it interesting in how gruesome and costly social signals can become, penile subincision is quite fitness-reducing (ejaculate flows out along the subincision), but there are so many things Australian aboriginals do that reduce fitness by a large amo