Cole Wyeth

LESSWRONG
LW

Cole Wyeth — LessWrong

Hmm, psychosecurity is an interesting reframing of epistemic rationality.

The front page looks beautiful on my laptop, particularly like that the post list can include images. The watercolor defaults are cool too. The top post is too prominent though.

Not as happy with the mobile experience. Rather than "top posts" I just have a "top post" which takes up basically the entire screen. Unfortunately, it's not very representative of my work and also has not held up very well. So it kind of sucks to have it totally dominate my profile page.

I can't find my karma anywhere and my bio is nearly hidden. It seems that it is long enough to somehow take up the space where my karma would be listed.

Since... (read more)

Replying toHave LLMs Generated Novel Insights?

Cole Wyeth1d

Have LLMs Generated Novel Insights?

I do, and I’m shifting in that direction.

Cole Wyeth3d

I don’t know, the impression I got was that he had a rather troubled life on and off the internet, for example “Ted loved his family dearly even while he struggled to connect.”

I’m sad to hear he didn’t quite live to see the arrival of real AGI.

Replying toCoping with Deconversion

Cole Wyeth3d

Coping with Deconversion

https://www.lesswrong.com/posts/x4dG4GhpZH2hgz59x/joy-in-the-merely-real

Replying toWhat's the concrete plan to become an incredibly agentic person?

Cole Wyeth8d

What's the concrete plan to become an incredibly agentic person?

Pursue mentorship from highly agentic people.

Cole Wyeth11d

Indeed in algorithmic information theory, the lower semicomputable semimeasures are an example of "subprobabilities." Much has been written about updating in this context.

Replying toStrong, bipartisan leadership for resistance to Trump.

Cole Wyeth18d

Strong, bipartisan leadership for resistance to Trump.

Your comment is unhelpful. I am pretty sure I do know what the post says, having recently read it.

The post focuses on independent institutions but the same principle applies to technocratic institutions. Otherwise I am not sure what you are getting at.

-1

At 12 pm EST today (Jan 26th), Marcus Hutter will be answering questions on his latest book (An Introduction to Universal Artificial Intelligence): https://uaiasi.com/2026/01/24/qa-with-marcus-hutter-at-the-final-meeting-of-the-iuai-reading-group/

Formally, this is the final meeting for the reading group, but feel free to drop in if you have read (most of) the book independently.

Who is responsible for shutting down rogue AI?

Cole Wyeth

1mo

A loss of control scenario would likely result in rogue AI replicating themselves across the internet, as discussed here: https://metr.org/blog/2024-11-12-rogue-replication-threat-model/

Under fast takeoff models, the first rogue AGI posing a serious takeover/extinction risk to humanity would very likely be the last, with no chance for serious opposition (e.g. Sable). This model seems theoretically compelling to me.

However, there is some recent empirical evidence that the basin of "roughly human" intelligence may not be trivial to escape. LLM agents seem increasingly competent and general, but continue to lag behind humans on long-term planning. If capabilities continue to develop in a highly jagged fashion, we may face rather dangerous rogue AI that still have some exploitable... (read 205 more words →)

It has been less than one year since I posted my model of what is going on with LLMs: https://www.lesswrong.com/posts/vvgND6aLjuDR6QzDF/my-model-of-what-is-going-on-with-llms

It already does not seem to hold up very well. I underestimated how far reasoning models based on CoT could scale. Generally, I’m now more convinced that labs can consistently convert money into capabilities by training transformers. In particular, I was previously very skeptical that RL would “just happen” to be solved with the same techniques as sequence prediction / text generation. It does seem that sequential decision making (eg Pokémon) continues to be the biggest weakness (by far relative to other capabilities), and agents are not very useful outside of coding, but... (read more)

•••

AIXI with general utility functions: "Value under ignorance in UAI"

Cole Wyeth

2mo

This updated version of my AGI 2025 paper with Marcus Hutter, "Value under ignorance in universal artificial intelligence," studies general utility functions for AIXI. Surprisingly, the (hyper)computability properties have connections to imprecise probability theory!

AIXI uses a defective Bayesian mixture called a semimeasure, which is often viewed as expressing a chance that the agent dies. I do not think that interpretation has been sufficiently justified. Recasting semimeasures as credal sets, we can recover the (recursive) value function from reinforcement learning for discounted nonnegative rewards. We can also obtain a wider class of lower semicomputable value functions, with optimal agents following a max min decision rule.

This is an early conference paper without complete proofs... (read 207 more words →)

It seems like we actually do not have a good name for the things that AI companies are building, weirdly enough.

LLMs? A hard-to-pronounce acronym for Large Language Models, which they were a couple of years ago, but now kind of aren't.

Frontier Models? Then how do we talk about models which are not frontier? Also, "models" of what? Also, does this term include AI systems which reach the frontier by a very different paradigm?

Reasoning Models? Better (best on this list?), though it seems to refer specifically to the models for math / coding problems, that use e.g. chain-of-thoughts, not the agents.

Models? I think this is what we are settling on? Seems fine, but... (read more)

A talk on embedded AIXI from Alexander Meulemans and Rajai Nasser is in 2 hours: https://uaiasi.com/2025/12/14/alexander-meulemans-rajai-nasser-on-embedded-aixi/

OpenAI claims 5.2 solved an open COLT problem with no assistance: https://openai.com/index/gpt-5-2-for-science-and-math/

This might be the first thing that meets my bar of autonomously having an original insight??

A good book gives your subagents something to fight about. A great book also helps them settle it.

Cognitive Tech from Algorithmic Information Theory

Cole Wyeth

2mo

Epistemic status: Compressed aphorisms.

This post contains no algorithmic information theory (AIT) exposition, only the rationality lessons that I (think I've) learned from studying AIT / AIXI for the last few years. Many of these are not direct translations of AIT theorems, but rather frames suggested by AIT. In some cases, they even fall outside of the subject entirely (particularly when the crisp perspective of AIT allows me to see the essentials of related areas).

Prequential Problem. The posterior predictive distribution screens off the posterior for sequence prediction, therefore it is easier to build a strong predictive model than to understand its ontology.

Reward Hypothesis (or Curse). Simple first-person objectives incentivize sophisticated but not-necessarily-intended intelligent... (read more)

My preference ordering over approaches to surviving the next decade, assuming I had unlimited political capital, is (don't build ASI and):

1: Solve mind uploading

2: ... or at least intelligence amplification

(before proceeding, and then maybe do)

3: Imitation learning (with radically improved theoretical foundations)

4: ... or corrigibility / oracles

(and don't)

5: Try to build aligned (or controlled) autonomous ASI agents

Unfortunately 5 seems to be profitable, in the short term, so I guess that's what we're doing. In fact, the "plan" seems to be the exact reverse or so of my preference ordering on what it should be.

I find the METR task length measurements very helpful for reasoning about timelines. However, they also seem like approximately the last benchmark I want the labs to be optimizing: high agency over long timescales, and also specifically focused on tasks relevant to recursive self-improvement. I’m sure that the frontier labs can (and do) run similar internal evals, but raising the salience of this metric (as a comparison point between models from different companies) seems risky. What has METR said about this?

“If you really believed X, you would do violence about it. Therefore, not X.”

I’ve seen this a few times with X := “AI extinction risk” (maybe fewer, seems like something I’d be prone to overestimate)

This argument is pretty infuriating because I do really believe X, but I’m obviously not doing violence about it. So it’s transparently false (to me) and the conversation is now about whether I’m being honest about my actual beliefs, which seems to undermine the purpose of having a conversation.

But it’s also kind of interesting, because it’s expressing a heuristic that seems valid - memes that incite “drastic” actions are dangerous, and activate a sort of epistemic immune system... (read more)

On the Aesthetic of Wizard Power

Cole Wyeth

2mo

Epistemic status: A response to @johnswentworth's "Orienting Towards Wizard Power." This post is about the aesthetic of wizard power, NOT its (nearest) instantiation in the real world, so that fictional evidence is appropriately treated as direct evidence.

Wentworth distinguishes the power of kings from the power of wizards. The power of kings includes authority, influence, control over others, and wealth. The power of wizards includes knowing how to make things like vaccines, a cure for aging, or the plumbing for a house. Indeed, these categories are described primarily by example (and of course by evocative naming). Wentworth argues that king power is not as real as wizard power; since it is partially about... (read 1241 more words →)

Embedded Universal Predictive Intelligence

Cole Wyeth

2mo

A team at Google has substantially advanced the theory of embedded agency with a grain of truth (GOT), including new developments on reflective oracles and an interesting alternative construction (the "Reflective Universal Inductor" or RUI).

(I was not involved in this work)

Abstract:

The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the

Cole Wyeth

3mo

Epistemic Status: Thoughts collected at the Sydney Mathematical Research Institute's Focus Period on the Mathematical Science of AI Safety, where I am a key participant. Not representative of anyone else's views.

So far Agent Foundations (AF) has made steady but perhaps slow progress which has not yielded currently practical AI safety ideas. However, engineering approaches (mechanistic interpretability, RLHF) have not proven robust, even for current models. One could imagine a world where frontier LLMs were not possible to jailbreak, and in that world it would be at least harder to argue that new alignment ideas are necessary. Therefore it is overdetermined that navigating the transition to artificial superintelligence (ASI) safely will require much... (read 2074 more words →)

Nontrivial pillars of IABIED

Cole Wyeth

4mo

Epistemic status: Jotting down some thoughts about the underlying Yudkowskian "doomer" philosophy, not a book review, intended for the lesswrong audience only.

I believe that one of the primary goals of the Sequences was to impart certain principles which illuminate the danger from unaligned AI. Most of these principles are widely accepted here (humans are not peak intelligence, some forth of the orthogonality thesis, etc.). However not everyone agrees with IABIED. I think there are a few additional nontrivial pillars of the worldview which can be taken as cruxes. This is an idiosyncratic list of the ones that occur to me. I think that all of them are correct, but if my model... (read 665 more words →)

•••

Cole Wyeth4mo

Semantics; it’s obviously not equivalent to physical violence.

I want an option to filter for writing with zero LLM influence.

I do not trust LLMs and I am not sure how I feel about LLM / human collaboration. As systems become more powerful, I am worried that they may become a serious epistemic hazard, up to and including actually hacking my brain. I would like to be able to protect myself from this aggressively.

For that reason, I think that the current LW policy on LLM usage is insufficient. Every post that uses an LLM in any part of its production process whatsover should be flagged as such. Personally, I am currently willing to accept some LLM usage upstream of the writing... (read more)

Alignment as uploading with more steps

Cole Wyeth

5mo

Epistemic status: This post removes epicycles from ARAD, resulting in an alignment plan which I think is better - though not as original, since @michaelcohen has advocated the same general direction (safety of imitation learning). However, the details of my suggested approach are substantially different. This post was inspired mainly by conversations with @abramdemski.

[Edit: The biggest obstacle to getting this proposal right is ensuring that the external world (beyond the human who should be imitated) is entirely screened off by the provided features and does not need to be predicted by the model. This will require learning to be performed in a very carefully constructed environment. I am not sure how difficult... (read 3903 more words →)

•••

Sleeping Experts in the (reflective) Solomonoff Prior

Daniel C

Daniel C, Cole Wyeth

5mo

Epistemic status: This first collaboration between Daniel Chiang (who is interested in the algorithmic information theory of incrementally constructed representations) and myself (Cole Wyeth) contains some fairly simple but elegant results that help illustrate differences between ordinary and reflective Oracle Solomonoff Induction.

Practical reasoning relies on context-specific prediction rules, which can be described as "incomplete models" or "partial experts." One simple example is "sleeping experts" which make predictions only at certain times (when they are "awake"), discussed here: https://www.lesswrong.com/posts/CGzAu8F3fii7gdgMC/a-simple-explanation-of-incomplete-models

Can Solomonoff's universal distribution take advantage of sleeping experts? We argue that the ordinary universal distribution has only a weak version of this property, while reflective Oracle Solomonoff Induction (rOSI) is perfectly capable of representing... (read 865 more words →)

New Paper on Reflective Oracles & Grain of Truth Problem

Cole Wyeth

6mo

This is a linkpost for https://www.arxiv.org/pdf/2508.16245

The grain of truth problem asks how multiple agents having consistent mental models can reason and learn about each other - recursively.

With Marcus Hutter, Jan Leike (@janleike), and Jessica Taylor (@jessicata) , I have revisited Leike et al.'s paper "A Formal Solution to the Grain of Truth Problem" (AFSGOTP) which studies games between reflective AIXI agents and... further formalized it.

The result is "Limit-Computable Grains of Truth for Arbitrary Computable Extensive-Form (Un)Known Games" (LCGOTACEFUG) which perhaps could have been called "A Formal Formal Solution to the Grain of Truth Problem." Our new paper has some new results, including:

An application to Self-AIXI which suggests an embedded version of AIXI.

... (read more)

Replying toAI 2027: What Superintelligence Looks Like

Cole Wyeth10mo

AI 2027: What Superintelligence Looks Like

I expect this to start not happening right away.

So at least we’ll see who’s right soon.

LESSWRONG
LW

LESSWRONG
LW

My model of what is going on with LLMs

Have LLMs Generated Novel Insights?

Cognitive Tech from Algorithmic Information Theory

Alignment as uploading with more steps

Cole Wyeth

Cole Wyeth

Who is responsible for shutting down rogue AI?

AIXI with general utility functions: "Value under ignorance in UAI"

Cognitive Tech from Algorithmic Information Theory

On the Aesthetic of Wizard Power

Embedded Universal Predictive Intelligence

Considering the Relevance of Computational Uncertainty for AI Safety

Nontrivial pillars of IABIED

I recklessly speculate about timelines

Meta-theory of rationality

AIXI Agent foundations

Deliberative Algorithms as Scaffolding

Cole Wyeth

My model of what is going on with LLMs

Have LLMs Generated Novel Insights?

Cognitive Tech from Algorithmic Information Theory

Alignment as uploading with more steps

Cole Wyeth

Cole Wyeth

Who is responsible for shutting down rogue AI?

AIXI with general utility functions: "Value under ignorance in UAI"

Cognitive Tech from Algorithmic Information Theory

On the Aesthetic of Wizard Power

Embedded Universal Predictive Intelligence

Considering the Relevance of Computational Uncertainty for AI Safety

Nontrivial pillars of IABIED

I recklessly speculate about timelines

Meta-theory of rationality

AIXI Agent foundations

Deliberative Algorithms as Scaffolding