Quick Takes

Over the last month or two I have rarely found it worth finishing any posts on the front page (even when I have done so, I usually felt ambivalent about the choice in retrospect). I am not sure whether I am simply busier with other things or whether the quality of the front page has went down, but as an incredibly avid lesswrong reader, it seems worth it for me to flag this bare fact (without proposing any particular explanation).

ControlAI has launched an official campaign in Canada, with 13 politicians across party lines backing a clear statement in favor of an international prohibition against developing superintelligent AI: https://controlai.com/canada-statement/en I used to be ambivalent about government intervention wrt AI safety, because politics can screw things up. However, it seems clearly positive to me now: the development of superintelligence impacts the whole human species, and as such, the democratic process should have a say in whether (and how) superintelligence is developed. Governments have been tolerant thus far, but the general public is not so happy about this, meaning that political campaigns have a good chance of success (in my estimation). Without such government intervention, the AI industry is on-track to develop superintelligence within the century if not the decade, and I do not believe safety research is ready to handle the risks. As Yudkowsky put it, only law can prevent extinction. Public reception of this campaign will make a big difference in the willingness of further politicians to endorse this or similar statements. Although I would normally hesitate to put signal-boost-style politics on LW, this could use a signal boost, and is quite relevant to the audience here.

One reflection I've had in the whole "AI use in the encyclical" affair is to slightly increase my trust in traditional media, especially non-American traditional media, and slightly decrease my trust in social media and new media epistemics. I tried my best to promote my analysis as legibly and reasonably as I could and focused on logos rather than ethos: I didn't frame my article with institutional affiliations and intentionally chose not to include obvious, flashy, but irrelevant signaling. Stuff I could've done but explicitly chose not to: get an ML professor to cosign my analysis, publish on Arxiv instead of Substack and LessWrong, highlight my past ML experience at Google, etc.  The traditional media response were somewhere between neutral and positive. There were some disappointments (eg a famous media org won't run the article without confirmation from a "primary source" -- aka the Vatican which is of course a non-starter). But mostly the traditional media just looked at the analysis and said "yeah looked reasonable." and either ran it as "Unconfirmed but seems right" or just "unconfirmed, period." Which seems fine. On the other hand, the social media attacks were very aggro. People just didn't seem to entertain it at all, many without reading it. And I got personally attacked a ton[1]. Also it wasn't picked up at all[2] by new media afaict (unless you count Russia Today as new media). This is exactly the type of story you might expect new media to be good for: story of institutional decay in the Old Guard, investigation by someone without credentials carefully laying out an epistemically rigorous and systematic case. And yet traditional media was happy to report it[3], social media kept yelling at me, and new media just got crickets. 1. ^ Don't worry, I didn't take it personally at all. Very much a "yapping of chihuahua in a tiny purse" moment. 2. ^ This is a slight exaggeration. Other than Russia Today a bunch of small AI news agg

Wanted to put these quotes on sign uncertainty by Holden Karnofsky in Oct 2025 here for my own future reference (h/t this post's authors), around the 4:11:30 mark, emphasis mine: [...] and slightly later, around the 4:18:30 mark, emphasis mine: [...] Illustrative pictorial version of Holden's sign uncertainty remarks, via RP's CCM tool Pictorial version of Holden's remark, from Rethink Priorities' cross-cause cost-effectiveness model, selecting "AI misalignment megaproject" and stacking the deck 70/30 in favor of the intervention conditional on having an effect instead of 51/49 (or more accurately 97.3/1.9/0.8 no effect / +ve / -ve, which I guess is in fact 51/49): I was raised in a culture that's probably closest to difference-making expected utility with moderate to high risk aversion for harming people and low but nonzero for animals and certain nonsentient things like the environment, which probably led to crystallised moral priors that make me instinctively find advice to be totally risk-neutral and EV-maxing come off as inhumanly detached with a whiff of sociopathic (including by well-meaning career advisors at orgs you've definitely heard of). RP's CCM tool puts illustrative numbers to this below, as part of their work to be useful to a wider variety of do-gooders: To my knowledge Open Phil (via Luke Muehlhauser) first talked about their sign uncertainty publicly in Dec 2020 in the context of explaining their AGI governance grantmaking so far, emphasis mine: [...] I had assumed that the information generated by those kinds of grants Luke mentioned, plus Luke himself specifically saying he'd gained some certainty by April 2023 in prefacing his tentative ideas for US AI policy [...] had substantively reduced Open Phil's sign uncertainty in their AGI grantmaking. But I guess not, going by Holden's "I tend to think it’s worse than 51/49" remark. Probably it's a mistake to round off OP's grantmaking as directionally reflecting the sign uncertainty of

i find it very interesting how becoming familiar with a place makes it feel so different, and yet it's recognizably the same place. especially when the first time you see the place you just think of it as a disconnected location floating around somewhere in abstract locationspace, but then slowly discover its relationship to other locations. sometimes this feels exhilarating, because the world feels more cohesive and whole and familiar. other times it's melancholy, because it feels like the magic has gone.

The Dead Economy Theory was on the top of Hacker News a few days ago. The post's strawmanning of rationalism/EA is a bit annoying, it's 58% AI-generated according to Pangram, and some of its ideas will already be well-known to LWers (despite claims that we have naively failed to consider them). However, the post packages together some potential blindspots I've been mulling over recently and brings up some points I hadn't thought about. Some ideas and takeaways from (or vaguely reminiscent of) the post: * Some people actually want a job - people find work to be a source of purpose and dignity, and may not be happy even with a solid AI-enabled UBI. * To support this point, the post cites research on "deaths of despair" in communities which lost industrial jobs and were ravaged by drug addiction. I didn't find this analogy super convincing; it seems like the people in these communities actually still had to work, just at less fulfilling, lower-paying jobs, which is a very different situation from getting a UBI that you could use to paint and write novels and so forth. * One might object: why can't we just have our super-smart AI set up a society that gives people something to do? AI could help build fulfilling communities, doing its best to accommodate those who don't want disruptive change, etc. I think it's becoming increasingly clear that even "ASI" may be bad at hard-to-evaluate conceptual tasks, like moral philosophy and social planning. * Even if those people who need work to keep themselves happy are given "jobs," I think there's some truth to "the dignity of work." Letting humans pretend to be useful despite not really being needed at all may not fulfill their values. * Although that logic seems a bit perverse... if we could have built the ASI to take your job but didn't, then in some sense you weren't really needed either way. But hey, I'm treating this as a butterfly idea, it doesn't need to make perfect sense right now. * More generally, mo

Something seems to be really wrong with Claude Opus 4.8. I like to test out new models on literary and poetic material. I used to send them the lyrics of Kate Bush's "The Kick Inside" and see what they could make of it, until they started just recognizing the song. Lately I've been using some text I wrote myself, with this prompt or something very close to it: [...] I'm not going to post the actual here. I want to keep using it with future LLMs. Who knows, I might even write the rest of the novel, and I hate teasers. If anybody really wants it, I can send it privately. The text is about 1500 words. It's a vignette, and its relation to the rest of the story won't be clear for a while. It's intentionally surreal-sounding, and it's not in trivially easy language. It drops lots of hints of different strengths about different things... as well as making some flat statements of fact. Most models do badly by human standards, but they get some of the themes, and make some reasonable, if rather conventional, guesses at what's going on. They do often tend to ignore critical phrases that matter out of proprotion to their length. They sometimes get the emphasis wrong. And the older and less sophisticated ones tend to lose facts. Opus 4.6 and 4.7 (and probably 4.5; I don't seem to have a record of trying it on this text) feel like interacting with a reader. Yes, they missed many of the hints, but so would a human. Opus 4.8 is lost, probably worse than Kimi K2. It gets some things about the mood, but that's about it. It seems to decide that certain phrases and paragraphs are ultra-salient, for no reason obvious to me. It confidently declares that a few things are central, then it mostly ignores the rest of the text. On the first run, it was so bad I thought the front end had mangled the file. Every run seems to fixate on the second to last paragraph, and some particular phrasing in it. The first run called it "the opening passage". Yet the very phrase the model most emph