LESSWRONG
LW

Mo Putera — LessWrong

To add to your point, Jacy Reese Anthis in Some Early History of Effective Altruism wrote

In general, EA emerged as the convergence from 2008 to 2012 at least 4 distinct but overlapping proto-EA communities, in order of founding:
The Singularity Institute (now known as Machine Intelligence Research Institute; MIRI) and the “rationalist” discussion forum LessWrong, founded by Eliezer Yudkowsky and others in 2000 and 2006
GiveWell, founded by Holden Karnofsky and Elie Hassenfeld in 2007, and Good Ventures, founded by with Dustin Moskovitz and Cari Tuna in 2011, which partnered together in 2014 as GiveWell Labs (now Open Philanthropy)
Felicifia, created by Seth Baum, Ryan Carey, and Sasha Cooper in 2008 as a utilitarianism discussion

... (read 926 more words →)

Mo Putera7d

From Nicholas Carlini's Anthropic blog post:

I tasked 16 agents with writing a Rust-based C compiler, from scratch, capable of compiling the Linux kernel. Over nearly 2,000 Claude Code sessions and $20,000 in API costs, the agent team produced a 100,000-line compiler that can build Linux 6.9 on x86, ARM, and RISC-V.

Bit more commentary on the capabilities benchmarking angle:

This project was designed as a capability benchmark. I am interested in stress-testing the limits of what LLMs can just barely achieve today in order to help us prepare for what models will reliably achieve in the future.
I’ve been using the C Compiler project as a benchmark across the entire Claude 4 model series. As

... (read 640 more words →)

Mo Putera9dQuick Take

When Scott posted Does age bring wisdom? 8 years ago, I read it and thought "will this happen to me?" These passages got burned into my impressionable young-ish brain:

I turn 33 today. I can only hope that age brings wisdom.
We’ve been talking recently about the high-level frames and heuristics that organize other concepts. They’re hard to transmit, and you have to rediscover them on your own, sometimes with the help of lots of different explanations and viewpoints (or one very good one). They’re not obviously apparent when you’re missing them; if you’re not ready for them, they just sound like platitudes and boring things you’ve already internalized.
Wisdom seems like the accumulation of

... (read 1045 more words →)

Mo Putera12d

FWIW the coauthor of the paper you linked provides more nuance here.

Mo Putera14d

Maybe one of Logan Strohl's Fucking Goddamn Basics of Rationalist Discourse? Actually the whole thing is short enough to repost here in its entirety to save the trouble of clicking through:

Don't say false shit omg this one's so basic what are you even doing. And to be perfectly fucking clear "false shit" includes exaggeration for dramatic effect. Exaggeration is just another way for shit to be false.
You do NOT (necessarily) know what you fucking saw. What you saw and what you thought about it are two different things. Keep them the fuck straight.
Performative overconfidence can go suck a bag of dicks. Tell us how sure you are, and don't pretend to know

... (read more)

Mo Putera14d

Yes please to the longer stab.

You also made me wonder what a shortlist of crucial considerations for digital minds ranked by something like instantiable mind-seconds "swing factor" (or whatever the more sophisticated version should be) would look like, where reversible computation's swing factor is 10^54 by Claude's lights.

Mo Putera16d

Meta: consider reposting on the EA Forum?

Julia Wise's 2013 Giving now vs. later: a summary still seems good today:

Reasons to give now:
You may get less altruistic as you age, so if you wait you may never actually donate.
Estimates of the returns on investment may be over-optimistic.
Giving to charities that can demonstrate their effectiveness provides an incentive for charities to get better at demonstrating that they're effective. We can't just wait for charities to improve — it takes donations to make that happen.
Having an active culture of giving encourages other people to give, too.
Better to eliminate problems as soon as possible. E.g. if we had eliminated smallpox in 1967 instead of 1977, many

... (read more)

Mo Putera26d

a skill which I respect in other people and which I aspire towards is noticing when other people are experiencing suffering due to violations of positive narratives, or fulfillment of negative narratives, and comforting them and helping nudge them back into a good narrative.

interesting, any examples online?

Replying toWhy has nuclear power been a flop?

Mo Putera1mo

Why has nuclear power been a flop?

I learned about ALARA from this post, so I wanted to note this here -- Reading List 01/17/2026 by Brian Potter at Construction Physics notes that ALARA may be taken down, maybe, not sure, can't find the memo to confirm:

Now it looks like the US Department of Energy is removing ALARA. Via E&E News:
Energy Secretary Chris Wright killed the Department of Energy’s decades-old radiation safety standard Friday.
Wright ended the department’s use of the As Low As Reasonably Achievable — or “ALARA” — principle, which has long been a staple of nuclear regulation. ALARA is rooted in the idea that any radiation exposure carries risks, but low doses can be justified by practical

... (read more)

Mo Putera1mo

Another whimsical "benchmark": Terry Tao wrote on Mathstodon that

There are now sufficiently many different examples of Erdos problems that have been resolved with various amounts of AI assistance and formal verification (see https://github.com/teorth/erdosproblems/wiki/AI-contributions-to-Erd%C5%91s-problems for a summary) that one can start to discern general trends.
Broadly speaking, we now see an empirical tradeoff between the level of AI involvement in the solution, and the difficulty or novelty of that solution. In particular, the recent solutions have spanned a spectrum roughly describable as follows:
1. Completely autonomous AI solutions to Erdos problems that are short and largely follow a standard technique. (In many, but not all, of these cases, some existing literature was found that proved

... (read 402 more words →)

Interesting quote on the downstream consequences of local speedup of output production by LLMs in business processes by Rafa Fernández, host of the Protocols for Business special interest group (SIG), from his essay Finding Fault Lines within the Firm:

AI is usually discussed in terms of automation or productivity. Those framings are not wrong, but they miss what makes AI adoption particularly revealing from a protocol perspective. While much of the public discussion frames AI in terms of cost-savings or new markets, our SIG has been focusing on the pressure it places on current coordination systems by changing the speed and scale at which work is produced ...
Across the SIG’s discussions, interviews, and

... (read 759 more words →)

Gemini 3 Pro analogized Scott Alexander to a beaver when I asked it to make sense of him, because "Scott is a keystone individual" and "in ecology, a keystone species (like the beaver) exerts influence disproportionate to its abundance because it creates the ecosystem in which others live":

He built the dam (The Community/Lighthaven) that pooled the water.
He signaled where the food was (Grants/Open Threads).
He warned of the predators (Moloch/AI Risk).

This was mildly funny. It was also striking how many factual details it erred in (the rest of the response that is, not the beaver analogy), which to an outsider might sound plausible if dramatic.

Just learned about the Templeton World Charity Foundation (TWCF), which is unusual in that one of their 7 core funding areas is, explicitly, 'genius':

Genius
TWCF supports work to identify and cultivate rare cognitive geniuses whose work can bring benefits to human civilization.
In this context, geniuses are not simply those who are classified as such by psychometric tests. Rather, they are those who: (1) generate significant mathematical, scientific, technological, and spiritual discoveries and inventions that benefit humanity or have the potential to transform human civilization, and (2) show exceptional cognitive ability, especially at an early age.
Eligible projects may include research on the benefits of various attributes of geniuses to humanity, biographical studies of individual

... (read 489 more words →)

Mildly funny analogy by John Cutler, niche audience, illustrating a failure mode that feels personally salient to me. Here's how it begins:

Imagine if a restaurant behaved like your average product team. The kitchen is packed. Everyone is moving. Every station is busy. Prep lists are long. Meetings are constant. There is always something to do. Chopping, rearranging, documenting, planning, replating.
But plates rarely reach customers. When they do, they’re late. Or wrong. Or cold. Or oddly disconnected from what the diners said they wanted. Yet the kitchen isn’t “failing,” exactly. It never looks like a crisis. No one storms out. No one flips a table. Diners don’t riot. They just lower their expectations

... (read more)

Eric Drexler's recent post on how concepts often "round to false" as they shed complexity and gain memetic fitness discusses a case study personal to him, that of atomically precise mass fabrication, which seems to describe a textbook cowpox-ing of doubt dynamic:

The history of the concept of atomically precise mass fabrication shows how rounding-to-false can derail an entire field of inquiry and block understanding of critical prospects.
The original proposal, developed through the 1980s and 1990s, explored prospects for using nanoscale machinery to guide chemical reactions by constraining molecular motions⁶. From a physics perspective, this isn’t exotic: Enzymes guide substrate molecules and provide favorable molecular environments to cause specific reactions; in molecular manufacturing,

... (read more)

A sad example of what Scott Aaronson called bureaucratic blankface: Hannah Cairo, who at 17 published a counterexample to the longstanding Mizohata-Takeuchi conjecture which electrified harmonic analysis experts the world over, decided after completing the proof to apply to 10 graduate programs. 6 rejected her because she didn't have a graduate degree nor a high school diploma (she'd been advised by Zvezdelina Stankova, founder of the top-tier Berkeley Math Circle, to skip undergrad at 14 and enrol straight in grad-level courses as she'd already taught herself an advanced undergrad curriculum by then from Khan Academy and textbooks). 2 admitted her but were then overridden by administrators. Only the U of Maryland and... (read 524 more words →)

•••

For my own future reference, here are some "benchmarks" (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical:

the AI village and blog, not really a "benchmark" per se but my richest source of intuitions about current frontier models' capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other "benchmark-equiv" frontier models
certain folks' opinions, e.g. lc on cybersec (like this take on AISLE), Kevin Buzzard and Terry Tao on math, Adam Karvonen on physical tasks relevant to manufacturing, Sarah Constantin on lit reviews, nostalgebraist on blog posts worth reading, Gwern and Jennifer Chen on writing diversity

... (read 617 more words →)

I wonder why the Claudes (Sonnet 3.7 and Opuses 4 and 4.1) are so much more reliably effective in the AI Village's open-ended long-horizon tasks than other labs' models.

when raising funds for charity, I recall seeing that Sonnet 3.7 raised ~90% of all funds (but I can no longer find donation breakdown figures so maybe memory confabulation...)
for the AI-organised event, both Sonnet 3.7 and Opus 4 sent out a lot more emails than say o3 and were just more useful throughout
in the merch store competition, the top 2 winners for both profits and T-shirt orders were Opus 4 and Sonnet 3.7 respectively, ahead of GhatGPT o3 and Gemini 2.5 Pro
- I can't

... (read more)

Unbundling Tools for Thought is an essay by Fernando Borretti I found via Gwern's comment which immediately resonated with me (emphasis mine):

I’ve written something like six or seven personal wikis over the past decade. It’s actually an incredibly advanced form of procrastination¹. At this point I’ve tried every possible design choice.
Lifecycle: I’ve built a few compiler-style wikis: plain-text files in a git repo statically compiled to HTML. I’ve built a couple using live servers with server-side rendering. The latest one is an API server with a React frontend.
Storage: I started with plain text files in a git repo, then moved to an SQLite database with a simple schema. The latest version is

... (read 1022 more words →)

Mo Putera's Shortform

Mo Putera

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

245

Non-loss of control AGI-related catastrophes are out of control too

Yi-Yang

Yi-Yang, Mo Putera, zeshen

For the Open Philanthropy AI Worldview Contest

Executive Summary

Conditional on AGI being developed by 2070, we estimate the probability that humanity will suffer an existential catastrophe due to loss of control (LoC) over an AGI system at ~6%, using a quantitative scenario model that attempts to systematically assess the relative probabilities of all AGI-driven catastrophes in a collectively exhaustive manner (details here; spreadsheet here).
We also estimate the overall probability that humanity will suffer an existential catastrophe due to an AGI-related catastrophe at ~12%. The remaining 12% - 6% = 6% of the probability mass derives from non-LoC AGI catastrophes (e.g. AI-exacerbated x-risks).
Given our estimate that p(existential catastrophe | non-LoC) ≈ p(existential catastrophe | LoC), we argue that Open

... (read 7116 more words →)

How should we think about the decision relevance of models estimating p(doom)?

Mo Putera

To illustrate what I mean, switching from p(doom) to timelines:

The recent post AGI Timelines in Governance: Different Strategies for Different Timeframes was useful to me in pushing back against Miles Brundage's argument that "timeline discourse might be overrated", by showing how choice of actions (in particular in the AI governance context) really does depend on whether we think that AGI will be developed in ~5-10 years or after that.
A separate takeaway of mine is that decision-relevant estimation "granularity" need not be that fine-grained, and in fact is not relevant beyond simply "before or after ~2030" (again in the AI governance context).
Finally, that post was useful to me in simply concretely specifying which

... (read 733 more words →)