Cole Wyeth — LessWrong

I am a PhD student in computer science at the University of Waterloo, supervised by Professor Ming Li and advised by Professor Marcus Hutter.

My current research is related to applications of algorithmic probability to sequential decision theory (universal artificial intelligence). Recently I have been trying to start a dialogue between the computational cognitive science and UAI communities. Sometimes I build robots, professionally or otherwise. Another hobby (and a personal favorite of my posts here) is the Sherlockian abduction master list, which is a crowdsourced project seeking to make "Sherlock Holmes" style inference feasible by compiling observational cues. Give it a read and see if you can contribute!

See my personal website colewyeth.com for an overview of my interests and work.

I do ~two types of writing, academic publications and (lesswrong) posts. With the former I try to be careful enough that I can stand by ~all (strong/central) claims in 10 years, usually by presenting a combination of theorems with rigorous proofs and only more conservative intuitive speculation. With the later, I try to learn enough by writing that I have changed my mind by the time I'm finished - and though I usually include an "epistemic status" to suggest my (final) degree of confidence before posting, the ensuing discussion often changes my mind again. As of mid-2025, I think that the chances of AGI in the next few years are high enough (though still <50%) that it’s best to focus on disseminating safety relevant research as rapidly as possible, so I’m focusing less on long-term goals like academic success and the associated incentives. That means most of my work will appear online in an unpolished form long before it is published.

Semantics; it’s obviously not equivalent to physical violence.

I expect this to start not happening right away.

So at least we’ll see who’s right soon.

Updates about LLM agency.

The AI 2027 forecast for mid-2025 scores on SWE-bench was not correct:

For example, we think coding agents will move towards functioning like Devin. We forecast that mid-2025 agents will score 85% on SWEBench-Verified.

(From the footnotes here.)

As of December 2025, the SOTA is around 81% for Claude 4.5 Opus, so this threshold probably will not be passed until 2026. Still, it does not seem far off.

Also, GPT-5.1-Codex-Max has a longer task length than I expected (perhaps because it is specifically for coding? But it seems there are always more tricks to maintain exponential growth - is this sustainable?).

On balance, I increasingly trust "straight lines" like METR task length to hold up in the short-medium term, simply because they have held up reliably without speeding up or slowing down (so perhaps I will lose my bet with @Daniel Kokotajlo). But even exponential growth is somewhat smooth, which seems consistent with my model's prediction that agency is hard. The evidence is (subjectively) weird - we are too ignorant about how LLMs work to make principled predictions. And I seem to have an unhealthy (awareness of my) reputation as lesswrong LLM skeptic, when in fact I am often confused and hold my beliefs on this rather weakly.

I don't see how this is relevant. I'm asking for examples of the OP's failed replications of safety papers which are popular on lesswrong. I am not disputing that ML papers often fail to replicate in general.

I don't understand why the OP would float the idea of founding an org to extend their work attempting replications, based on the claim that replication failures are common here, without giving any examples (preferably examples that they personally found). This post is (to me) indistinguishable from noise.

Downvoted, making this claim without examples is inexplicable.

It’s a continuous probability measure, meaning it has no atoms, but it does assign positive probability to all cylinder sets. If you take the binary representation of reals in the [0,1] interval, P_u comes from the Lebesgue measure (a uniform distribution).

The difference is not a “practical” one as long as you only use the posterior predictive distribution, but in some AIXI variants (KSA, certain safety proposals) the posterior weights themselves are accessed and the form may matter. Arguably this is a defect of those variants.

I think that I need this in my variant of AIXI in order to filter out "world models" which don't necessarily halt, and I think this will be enough to do so, but I'll leave working out the details to a later post.

AIXI's hypothesis class is the (lower semicomputable chronological) semimeasures, so I do not know why halting should be required for this application?

(My question is now of mainly historical interest, since the later versions of reflective oracles do not require queries to be about machines that halt, AND because they "complete" the distributions generated by those machines to proper measures)

Yes, they’re not consistently highly correlated, my guess could be wrong.

LESSWRONG
LW

LESSWRONG
LW

Sequences

Posts

Wikitag Contributions

Comments