LessWrong

LESSWRONG
LW

LessWrong

The resurrected LessWrong Power Reader is now on GitHub. Thought I'd post it here in a lowkey way in case you (or anyone else) want to test it or review its API usage, make sure it's not doing something undesirable, before I make an announcement post.

the current intro and feature list

A fast, context-first reader for LessWrong and the EA Forum, designed to make high-volume reading and thread navigation feel effortless.

Chronological Reader Core: Shows comments in strict time order with date-based pagination, so you can read without gaps.
Deep Thread Context: Loads missing parents and replies so deep comments still make sense.
Post + Comment Power Actions: Inline controls to expand/load post bodies, load all

... (read more)

Wei Dai20hQuick Take

My 6 years as a trader / active investor

The Dilbert Afterlife by Scott Alexander, Jan 16, 2026:

Michael Jordan was the world’s best basketball player, and insisted on testing himself against baseball, where he failed. Herbert Hoover was one of the world’s best businessmen, and insisted on testing himself against politics, where he crashed and burned. We’re all inmates in prisons of different names. Most of us accept it and get on with our lives. Adams couldn’t stop rattling the bars.

The EMH Aten't Dead by Richard Meadows, May 15, 2020:

Which only leaves the initial claim that "at least for me this puts a final nail in the coffin of EMH."
This is a polite

... (read 676 more words →)

Wei Dai4d

Suppose we rule out pure CDT. That still leaves "whatever the right DT is (even if it's something like FDT/UDT), if you actually run the math on it, it says that rewarding people after the fact for one-time actions provides practically zero incentives (if people means pre-singularity humans)". I don't see how we can confidently rule this out.

Wei Dai4d

In my personal vibe-coding projects, I'm reviewing ~0% of code, but having to do a lot more testing (relative to writing the code myself) because the AI is constantly introducing regressions (breaking what previously worked), which are not being caught by its test code because either the spec wasn't detailed enough to cover every possibility or edge case (i.e., the AI can't read between the lines to figure out what I want without being written down in detail, or it doesn't care), or its testing code isn't good enough to catch a lot of bugs.

As an example, when it adds a new UI element, the styling would often be inconsistent with other nearby elements, and I'd have to tell it to add a requirement to the spec that this set of elements should have consistent styling.

If others have similar experiences, we'll still have "% testing done by humans" metric to descend after "% of code reviewed by humans" goes to 0.

Wei Dai4d

But also, beyond all of that, the arguments around decision-theory are I think just true in the kind of boring way that physical facts about the world are true, and saying that people will have the wrong decision-theory in the future sounds to me about as mistaken as saying that lots of people will disbelieve the theory of evolution in the future. It's clearly the kind of thing you update on as you get smarter.

This seems way overconfident:

Our current DT candidates (FDT/UDT) may not be correct or on the right track.
Smart != philosophically competent. (There are literally millions of people in China with higher IQ than me, but AFAIK none of them

... (read more)

Wei Dai6d

It confused me that Opus 4.6's System Card claimed less verbalized evaluation awareness versus 4.5:

On our verbalized evaluation awareness metric, which we take as an indicator of potential risks to the soundness of the evaluation, we saw improvement relative to Opus 4.5.

but I never heard about Opus 4.5 being too evaluation aware to evaluate. It looks like Apollo simply wasn't part of Opus 4.5's alignment evaluation (4.5's System Card doesn't mention them).

This probably seems unfair/unfortunate from Anthropic's perspective, i.e., they believe their models are becoming less eval aware, but due to Apollo's conclusions being spread on social media, a lot of people probably got the impression that models are getting more eval aware. Personally I'm not sure we can trust Anthropic's verbalized evaluation awareness metric, and wish Apollo had done evals on 4.5 too to give us an external comparison.

Wei Dai7d

It looks like direct xAI/Grok support was only added to OpenClaw 8 hours ago in this commit and still unreleased. You could have used Grok with it via OpenRouter, but I doubt this made up a significant fraction of Clawdbot/Moltbot/OpenClaw agents.

Perplexity estimates the model breakdown as:

Anthropic Claude ~85%
Local (Ollama/vLLM) ~10%
OpenAI / Google ~5%

-1

Replying toBuilding AIs that do human-like philosophy

Wei Dai7d

Building AIs that do human-like philosophy

What happens if down the line we have AIs that are more competent than humans in most areas, but lag behind or are distorted in philosophy? Seems like it would be too late to pause/stop the AI transition at that point.

I subsequently realized that there is actually a narrow path from this situation to an AI pause/stop, which requires that the AIs themselves realize both that they're bad at philosophy and the strategic implications of this. This is a form of strategic competence that I think AIs will probably also lack by default (by the time AIs are more competent than humans in most areas including being able to create next generation AIs), but may be somewhat easier to fix than philosophical incompetence.

Wei Dai7dQuick Take

The striking contrast between Jan Leike, Jan 22, 2026:

Our current best overall assessment for how aligned models are is automated auditing. We prompt an auditing agent with a scenario to investigate: e.g. a dark web shopping assistant or an imminent shutdown unless humans are harmed. The auditing agent tries to get the target LLM (i.e. the production LLM we’re trying to align) to behave misaligned, and the resulting trajectory is evaluated by a separate judge LLM. Albeit very imperfect, this is the best alignment metric we have to date, and it has been quite useful in guiding our alignment mitigations work.
[...]
But the most important lesson is that simple interventions are very effective

Wei Dai

10d

If AIs became strategically competent enough, they may realize that RSI is too dangerous because they're not good enough at alignment or philosophy or strategy, and potentially convince, help, or work with humans to implement an AI pause. This presents an alternative "victory condition" that someone could pursue (e.g. by working on AI strategic competence) if they were relatively confident about the alignment of near-human-level AIs but concerned about the AI transition as a whole, for example because they're worried about alignment of ASI, or worried about correctly solving other philosophical problems that would arise during the transition. (But note that if the near-human-level AIs are not aligned, then this effort could... (read 287 more words →)

Replying toSolving adversarial attacks in computer vision as a baby version of general AI alignment

Wei Dai10d

Solving adversarial attacks in computer vision as a baby version of general AI alignment

We can view the problem as a proving ground for ideas and techniques to be later applied to the AI alignment problem at large.

Do you have any examples of such ideas and techniques? Are any of the ideas and techniques in your paper potentially applicable to general AI alignment?

My attempt to resurrect the old LW Power Reader is facing an obstacle just before the finish line, due to current LW's API limitations. So this is a public appeal to the site admins/devs to relax the limit.

Specifically, my old code relied on LW1 allowing it to fetch all comments posted after a given comment ID, but I can't find anything similar in the current API. I tried reproducing this by using the allRecentComments endpoint in GraphQL, but due to the offset parameter being limited to <2000, I can't fetch comments older than a few weeks. The Power Reader is part designed to allow someone to catch up on or skim weeks/months... (read more)

In retrospect it seems like such a fluke that decision theory in general and UDT in particular became a central concern in AI safety. In most possible worlds (with something like humans) there is probably no Eliezer-like figure, or the Eliezer-like figure isn't particularly interested in decision theory as a central part of AI safety, or doesn't like UDT in particular. I infer this from the fact that where Eliezer's influence is low (e.g. AI labs like Anthropic and OpenAI) there seems little interest in decision theory in connection with AI safety (cf Dario Amodei's recent article which triggered this reflection), and in other places interested in decision theory, that aren't downstream of Eliezer popularizing it, like academic philosophy, there's little interest in UDT.

If this is right, it's another piece of inexplicable personal "luck" from my perspective, i.e., why am I experiencing a rare timeline where I got this recognition/status.

•••

Possible root causes if we don't end up having a good long term future (i.e., realize most of the potential value of the universe), with illustrative examples:

Technical incompetence
- We fail to correctly solve technical problems in AI alignment.
- We fail to build or become any kind of superintelligence.
- We fail to colonize the universe.
Philosophical incompetence
- We fail to solve philosophical problems in AI alignment
- We end up optimizing the universe for wrong values.
Strategic incompetence
- It is not impossible to cooperate/coordinate, but we fail to figure out how.
- We fail to have other important strategic insights
  - E.g., related to whether it's better in the long run to build AGI first, or enhance human intelligence first
- We have the insights but fail to

... (read more)

"Utility" literally means usefulness, in other words instrumental value, but in decision theory and related fields like economics and AI alignment, it (as part of "utility function") is now associated with terminal/intrinsic value, almost the opposite thing (apparently through some quite convoluted history). Somehow this irony only occurred to me ~3 decades after learning about utility functions.

My 2003 Post on the Evolutionary Argument for AI Misalignment

Wei Dai

1mo

This was posted to SL4 on the last day of 2003. I had largely forgotten about it until I saw the LW Wiki reference it under Mesa Optimization^[1]. Besides the reward hacking angle, which is now well-trodden, it gave an argument based on the relationship between philosophy, memetics, and alignment, which has been much less discussed (including in current discussions about human fertility decline), and perhaps still worth reading/thinking about. Overall, the post seems to have aged well, aside from the very last paragraph.

For historical context, Eliezer had coined "Friendly AI" in Creating Friendly AI 1.0 in June 2001. Although most of it was very hard to understand and subsequently disavowed by... (read 429 more words →)

A Conflict Between AI Alignment and Philosophical Competence

Wei Dai

2mo

(This argument reduces my hope that we will have AIs that are both aligned with humans in some sense and also highly philosophically competent, which aside from achieving a durable AI pause, has been my main hope for how the future turns out well. As this is a recent realization^[1], I'm still pretty uncertain how much I should update based on it, or what its full implications are.)

Being a good alignment researcher seems to require a correct understanding of the nature of values. However metaethics is currently an unsolved problem, with all proposed solutions having flawed or inconclusive arguments, and lots of disagreement among philosophers and alignment researchers, therefore the current meta-correct... (read 485 more words →)

Relitigating the Race to Build Friendly AI

Wei Dai

2mo

Recently I've been relitigating some of my old debates with Eliezer, to right the historical wrongs. Err, I mean to improve the AI x-risk community's strategic stance. (Relevant to my recent theme of humans being bad at strategy—why didn't I do this sooner?)

Of course the most central old debate was over whether MIRI's circa 2013 plan, to build a world-altering Friendly AI^[1], was a good one. If someone were to defend it today, I imagine their main argument would be that back then, there was no way to know how hard solving Friendliness/alignment would be, so it was worth a try in case it turned out to be easy. This may seem... (read 723 more words →)

This is (approximately) my forum.

I was curious what Habryka meant when he said this. Don't non-profits usually have some kind of board oversight? It turns out (from documents filed with the State of California), that Lightcone Infrastructure, which operates LW, is what's known as a sole-member nonprofit, with a 1-3 person board of directors determined by a single person (member), namely Oliver Habryka. (Edit: My intended meaning here is that this isn't just a historical fact, but Habryka still has this unilateral power. And after some debate in the comments, it looks like this is correct after all, but was unintentional. See Habryka's clarification.)

However, it also looks like the LW domain is... (read more)

Having finally experienced the LW author moderation system firsthand by being banned from an author's posts, I want to make two arguments against it that may have been overlooked: the heavy psychological cost inflicted on a commenter like me, and a structural reason why the site admins are likely to underweight this harm and its downstream consequences.

(Edit: To prevent a possible misunderstanding, this is not meant to be a complaint about Tsvi, but about the LW system. I understand that he was just doing what he thought the LW system expected him to do. I'm actually kind of grateful to Tsvi to let me understand viscerally what it feels like to be... (read more)

•••

Please, Don't Roll Your Own Metaethics

Wei Dai

3mo

One day, when I was an intern at the cryptography research department of a large software company, my boss handed me an assignment to break a pseudorandom number generator passed to us for review. Someone in another department invented it and planned to use it in their product, and wanted us to take a look first. This person must have had a lot of political clout or was especially confident in himself, because he rejected the standard advice that anything an amateur comes up with is very likely to be insecure and he should instead use one of the established, off the shelf cryptographic algorithms, that have survived extensive cryptanalysis (code breaking)... (read 506 more words →)

154

An update on this 2010 position of mine, which seems to have become conventional wisdom on LW:

In my posts, I've argued that indexical uncertainty like this shouldn't be represented using probabilities. Instead, I suggest that you consider yourself to be all of the many copies of you, i.e., both the ones in the ancestor simulations and the one in 2010, making decisions for all of them. Depending on your preferences, you might consider the consequences of the decisions of the copy in 2010 to be the most important and far-reaching, and therefore act mostly as if that was the only copy. [Emphasis added]

In the subsequent 15 years, I've upweighted influencing the multiverse... (read more)

Problems I've Tried to Legibilize

Wei Dai

3mo

Looking back, it appears that much of my intellectual output could be described as legibilizing work, or trying to make certain problems in AI risk more legible to myself and others. I've organized the relevant posts and comments into the following list, which can also serve as a partial guide to problems that may need to be further legibilized, especially beyond LW/rationalists, to AI researchers, funders, company leaders, government policymakers, their advisors (including future AI advisors), and the general public.

Philosophical problems
1. Probability theory
2. Decision theory
3. Beyond astronomical waste (possibility of influencing vastly larger universes beyond our own)
4. Interaction between bargaining and logical uncertainty
5. Metaethics
6. Metaphilosophy: 1, 2
Problems with specific philosophical and alignment ideas
1. Utilitarianism: 1, 2
2. Solomonoff induction
3. "Provable" safety
4. CEV
5. Corrigibility
6. IDA (and

... (read 366 more words →)

137

The Inhumanity of AI Safety

A: Hey, I just learned about this idea of artificial superintelligence. With it, we can achieve incredible material abundance with no further human effort!

B: Thanks for telling me! After a long slog and incredible effort, I'm now a published AI researcher!

A: No wait! Don't work on AI capabilities, that's actually negative EV!

B: What?! Ok, fine, at huge personal cost, I've switched to AI safety.

A: No! The problem you chose is too legible!

B: WTF! Alright you win, I'll give up my sunken costs yet again, and pick something illegible. Happy now?

A: No wait, stop! Someone just succeeded in making that problem legible!

B: !!!

Legible vs. Illegible AI Safety Problems

Wei Dai

3mo

Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to). But some problems are illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved. (Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)

From an x-risk perspective, working on highly legible safety problems has low... (read 420 more words →)

370

•••

Trying to understand my own cognitive edge

Wei Dai

3mo

I applaud Eliezer for trying to make himself redundant, and think it's something every intellectually successful person should spend some time and effort on. I've been trying to understand my own "edge" or "moat", or cognitive traits that are responsible for whatever success I've had, in the hope of finding a way to reproduce it in others, but I'm having trouble understanding a part of it, and try to describe my puzzle here. For context, here's an earlier EAF comment explaining my history/background and what I do understand about how my cognition differs from others.^[1]

More Background

In terms of raw intelligence, I think I'm smart but not world-class. My SAT was only 1440,... (read 1169 more words →)

Managing risks while trying to do good

Wei Dai

I often think about "the road to hell is paved with good intentions".^[1] I'm unsure to what degree this is true, but it does seem that people trying to do good have caused more negative consequences in aggregate than one might naively expect.^[2] "Power corrupts" and "power-seekers using altruism as an excuse to gain power" are two often cited reasons for this, but I think don't explain all of it.

A more subtle reason is that even when people are genuinely trying to do good, they're not entirely aligned with goodness. Status-seeking is a powerful motivation for almost all humans, including altruists, and we frequently award social status to people for merely trying to do... (read 330 more words →)

LESSWRONG
LW

LESSWRONG
LW

Wei Dai

Legible vs. Illegible AI Safety Problems

A tale from Communist China

Morality is Scary

UDT shows that decision theory is more puzzling than ever

Wei Dai

Wei Dai

Increasing AI Strategic Competence as a Safety Approach

My 2003 Post on the Evolutionary Argument for AI Misalignment

A Conflict Between AI Alignment and Philosophical Competence

Relitigating the Race to Build Friendly AI

Please, Don't Roll Your Own Metaethics

Problems I've Tried to Legibilize

Legible vs. Illegible AI Safety Problems

Wei Dai

Legible vs. Illegible AI Safety Problems

A tale from Communist China

Morality is Scary

UDT shows that decision theory is more puzzling than ever

Wei Dai

Wei Dai

Increasing AI Strategic Competence as a Safety Approach

My 2003 Post on the Evolutionary Argument for AI Misalignment

A Conflict Between AI Alignment and Philosophical Competence

Relitigating the Race to Build Friendly AI

Please, Don't Roll Your Own Metaethics

Problems I've Tried to Legibilize

Legible vs. Illegible AI Safety Problems

My 6 years as a trader / active investor

More Background