Quick Takes

do you have an ambitious idea for how to make AGI go well? do you need money? do you hate bureaucracy and friction? apply now for microgrants! please read the entire doc before applying. only send applications to the designated location, or they will be automatically rejected. https://docs.google.com/document/d/10zAp2bXTkZgiPreIm4crp38TFco4KleFN14Kw5BprAs/edit?tab=t.0

Jamie Joyce seems like a better than replacement candiate for the 12th California Congressional district Jamie Joyce seems like a better than replacement candidate for the 12th California Congressional district, which includes Oakland and Berkeley. She came second in the primary, meaning that she will be in the ballot in November. Here is her Ballotpedia page. I know her from the FLF fellowship last year. She is not a candidate which embodies rationalist mores, but is rather more bubbly and normal. However, worries about AI are at or near the top of her policy agenda (framed as a proposed act of Congress named the “MAD act”), and she is generally able to integrate worries about AI safety with other mainstream issues. That is, she is not a rationalist candidate, but is one cadidate which rationalists could form part of a coalition to elect. Her opponent is a very mainstream & uninspiring Democratic politician. Unlike Alex Bores’ campaign, which is far away in NYC, California’s 12th Congressional district is right in the seat of the rationalists’ power. This means that acts like canvassing, putting up posters, etc. would be much cheaper to do. Shooting from the hip, maybe I’d give her a 10% chance of success on her own, and a 20% chance if the rats meaningfully back her (e.g., she gets 30% as much effort thrown into her campaign as the Alex Bores has). But I haven’t thought deeply about these probabilities. Doing some quick math, if there are 2k to 10k rationalists in Berkeley, and they each can convince 2 to 10 friends, that’s 10k to 70k votes. This would be a big chunk of the overall population of 750K inhabitants, and a pretty big chunk of the 150K-190K votes one would have needed to win the 2024 election for that same district. So again, rationalists alone would not be able to win the election, but could as part of a coalition. Per the Ballotpedia page, Jamie raised $14K so far and spent 10k on the primary, winning 28K votes (!!), meaning getting a vote for

would people find an alignment microgrant program useful? handing out grants on the order of $10k with very minimal bureaucracy - a 15 minute application process and a 30 minute retrospective call 3 months down the line. is this a quantity of money that enough to matter at all? is overhead around grant applications actually a big issue? is there really good work that nobody is funding already?

notes! I've just posted a repo with a bunch of my notes from 2023–2026, mostly on topics with relevance to AI alignment; see here for more meta information. An assortment of 101 items from the vault: * AI safety presentation slides (or really self-contained slideuments): * Variants of the alignment problem (at the AFFINE Superintelligence Alignment Seminar) * Verification-based alignment schemes (at the AFFINE Superintelligence Alignment Seminar) * Inspection of thinking as a source of ideas for understanding minds (at the AFFINE Superintelligence Alignment Seminar) * Model-wise thinking (at the AFFINE Superintelligence Alignment Seminar) * Intro to AGI safety (at the University of Tartu) * Impact cases for guardrails/monitoring/verification (at Mila) * Why fear a math AI? (at Mila) * on understanding: * good concepts are concepts that support inferences * predicting specific things * model-thinking in mathematics * how categoricity (or universal properties) relate to the implementation of mental structures * gaining the integral notion * cheap inferences * turning activity-situations into abstract games * a stylized genealogy of sonic concepts * condensation: * condensation coefficients from scale-freeness * condensation coefficients compatible with the loss being a positive combination of subset losses * linear condensation * limitations of condensation * condensation coding * valuing, ethics, metaethics: * mundane malignity * it is natural for a mind to remake the world * terminality causes unmooredness * seeing (meta)ethics clearer via mathematical logic * centering ethics on coming up with ways to act * there are many good and important value-flavored structures * the epistemics-ethics bridge * in one sense, human values are very complicated; in another sense, human values are very simple * argument against universal value extraction device * an illustrat

Why is there no talk about GLM 5.2? It's a Chinese open weights model released June 13. Better than Gemini, Claude Sonnet, and Grok according to many benchmarks. E.g. on artificialanalysis.ai and on arena.ai/leaderboard. On frontierswe.com it even beats GPT 5.5, second to only Claude. LiveBench ranks it number 1 for its "agentic coding" measure. It's not just open weights but a little open about the methods it used, and is less than 1 trillion parameters. There's no mention on LessWrong, little mention on Reddit, and no mainstream results on Google search. Why? Is it still too early? Or am I just being tricked by the benchmaxxing? But I saw a demo of code it wrote (see youtube.com/watch?v=6d__WOpZswY) which looked incredibly impressive, it feels like it's not just benchmaxxing because it's pretty decent across the board. Or did Z.ai just screw up their marketing by versioning it as GLM 5.2 instead of GLM Mythical Fable Pro 6???[1] 1. ^ Since GLM 5.1 is far below than GLM 5.2, being behind other open weights models like DeepSeek, MiniMax, Kimi and MiMo

The AI safety ecosystem is so well resourced that it has been correctly identified by many as one of the best paths into high prestige AI research jobs. This person on twitter has written a popular article about getting into frontier ai labs and a Field Guide to AI Fellowships. The "AI Fellowships" are mostly AI safety programs funded by CG/OpenPhil. I have also noticed that people in ML research are quite likely to have heard of MATS and be interested in participating, even when they have very little interest in AI safety. Idk how good or bad this is. It definitely causes a lot of ML researchers to engage with the AI safety literature, where they otherwise would not have. But it's worth noting that while the primary driver of applications to programs like MATS used be concern about AI safety, now it is increasingly a desire to work at a frontier lab. I used to select participants for LASR labs, one of programs listed, and we actively tried to choose people who cared about AI safety, but I think we often did not succeed and indeed some people on the program now work at frontier labs in roles that I think have little to do with safety. Safety-washing in practice mostly looks like people rationalizing research or jobs that they would have done anyway as necessary for safety. It's easy to do because it is in fact genuinely unclear in most cases what is helpful and harmful for safety. Only by looking at the larger pattern can we notice the suspicious abundance of conveniently overlapping opportunities that further both safety and a person's short-term interests. However, I think it will be less common in people whose primary motivation is to prevent AI catastrophe.

One approach to automating AI safety research: 1. Scrape LessWrong for AI safety experiment ideas 2. Tell your favorite AI to implement everything that looks doable 3. Profit (in the sweet currency of impact) Many people have posted ideas about AI safety on LW. But not many competent researchers want to read through other people's random experiment ideas, figure out whether they're any good, and implement them, when they could be working on their own ideas. We may be able to pick a lot of low-hanging fruit by making AIs do that work. Could we try this plan with, say, Claude Mythos? I'm not sure. One major obstacle is taste: the AI has to independently implement a high-level experiment idea, and even proactively modify the original idea when it notices flaws. Also, once it's run thousands of different experiments, it needs to understand which results are most exciting and worth showing to a human, and make the reasons for its excitement legible enough for the human to verify. This kind of taste might not be strictly necessary for capabilities research, which is mostly about hill-climbing on benchmarks, but it's critical for fuzzy, conceptual tasks like AI safety research. What would we need to do in order to point an AI at a giant pile of vague, underspecified AI safety research ideas and feel confident we'd get something useful out of it?[1] Partly for this reason, I'd encourage people to release any AI-safety-related ideas they're sitting on. (H/t to @Kaarel for publishing their own notes and inspiring this post.) I've been wanting to polish and write up several ideas myself, but in the meantime, I just made my "AI safety ideas" Google Doc public. While I can't promise my personal notes will make sense to anyone else, any interested humans or AIs should feel free to take a look :) 1. ^ Excluding capabilities improvements that we expect to happen soon by default, which safety-focused people should probably not work on.