ozziegooen

LESSWRONG
LW

ozziegooen — LessWrong

Replying toAre there lessons from high-reliability engineering for AGI safety?

Are there lessons from high-reliability engineering for AGI safety?

I imagine we'd both agree that there can and should be a lot of evals and attempts at robustness / reliability for small / low-level systems?

It seems like the disagreement is in how useful such work will be for critical and broader alignment challenges.

Replying toAre there lessons from high-reliability engineering for AGI safety?

ozziegooen8d

Are there lessons from high-reliability engineering for AGI safety?

I appreciate that you're engaging in this question, but I have a hard time taking much away from this.

Quickly:

1. I think we're both probably skeptical of what Joshua Achiam is approaching. My hunch is that he's representing a particularly niche view on things.

2. It feels to me like there's a gigantic challenge of choosing what scale to approach this problem. You seem to be looking at AGI and it's effects from a very high-level. Like, we need to run evals on massive agents exploring the real world - and if this is too complex for more regular robust engineering, then we better throw out the idea of robust engineering. To me this... (read more)

Replying toConditional Kickstarter for the "Don't Build It" March

ozziegooen12d

Conditional Kickstarter for the "Don't Build It" March

Went here to post something similar. My quick guess is that such a march would get a better fit of attendance times direction by expanding the scope a fair bit, but I realize that the creators have certain preferences here.

Replying toWhat's Going on With OpenAI's Messaging?

ozziegooen12d

What's Going on With OpenAI's Messaging?

Update:

I've recently been experimenting with a small llm-generated wiki, mostly with Claude Code.

I was curious about the above, so I had it generate a page to much better cover Sam Altman's history, and his prediction track record.

https://www.longtermwiki.com/knowledge-base/people/sam-altman/
https://www.longtermwiki.com/knowledge-base/people/track-records/sam-altman-predictions/

I don't think there's anything incredibly novel there, but I think it's a decent overview.

I think the main Sam Altman page is more interesting than the prediction page, because his actions were a lot more interesting/informative than his predictions.

Replying toAnnouncing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen2mo

Announcing RoastMyPost: LLMs Eval Blog Posts and More

Thanks for letting me know!

I'm not very attached to the name. Some people seem to like it a lot, some dislike.

I don't feel very confident in how it will be most used 4-12 months from now. So my plan at this point is to wait on use, then consider renaming later as the situation gets clearer.

Opinion Fuzzing: A Proposal for Reducing & Exploring Variance in LLM Judgments Via Sampling

ozziegooen

2mo

Summary
LLM outputs vary substantially across models, prompts, and simulated perspectives. I propose "opinion fuzzing" for systematically sampling across these dimensions to quantify and understand this variance. The concept is simple, but making it practically usable will require thoughtful tooling. In this piece I discuss what opinion fuzzing could be and show a simple example in a test application.

LLM Use
Claude Opus rewrote much of this document, mostly from earlier drafts. It also did background research, helping with the citations.

Introduction

LLMs produce inconsistent outputs. The same model with identical inputs will sometimes give different answers. Small prompt changes produce surprisingly large output shifts.[1] If we want to use LLMs for anything resembling reliable judgments (research evaluation,... (read 1316 more words →)

Replying toAnnouncing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen2mo

Announcing RoastMyPost: LLMs Eval Blog Posts and More

Thanks!

I tried adding a "Clarity Coach" earlier. I think that this is a sort of area where RoastMyPost probably wouldn't have a massive advantage over custom LLM prompts directly, but we might be able to make some things easy.

It would be very doable to add custom evaluators to give tips along these lines. Doing a great job would likely involve a fair bit of prompting, evaluation, and iteration. I might give it a stab and if so will get back to you on this.

(One plus is that I'd assume this would be parallelizable, so it could be fast at least)

•••

Replying toAnnouncing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen2mo

Announcing RoastMyPost: LLMs Eval Blog Posts and More

Thanks for reporting your findings!

As I stated here, the Fact Checker has a bunch of false positives, and you've noted some.

The Fact Checker (and other checkers) have trouble telling which claims are genuine and which are part of fictional scenarios, a la AI-2027.

The Fallacy Checker is overzealous, and doesn't use web search (adds costs), so will particularly make mistakes when it's above the date the models were trained.

There's clearly more work to do to make better evals. Right now I recommend using this as a way to flag potential errors, and feel free to add any specific evaluator AIs that you think would be a fit for certain documents.

Replying toAnnouncing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen2mo

Announcing RoastMyPost: LLMs Eval Blog Posts and More

I experimented with Opus 4.5 a bit for the Fallacy Check. Results did seem a bit better, but costs were much higher.

I think the main way I could picture adding money is to add some agentic setup that does a deep review of a certain paper and presents a summary. I could see the marginal costs of this being maybe $10 to $50 per 5k words or so, using a top model like Opus. That said, the fixed costs of doing a decent job seem frustrating, especially because we're still lacking easy API use of existing agents (My preferred method would be a high-level Claude Code API, but that doesn't really exist yet).

I've been thinking of having competitions here, for people to make their own reviews, then we could compare with a few researchers and LLMs. I think this area could make for a lot of cleverness and innovation.

Replying toAnnouncing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen2mo

Announcing RoastMyPost: LLMs Eval Blog Posts and More

I also added a table back to this post that gives a better summary.

Replying toAnnouncing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen2mo

Announcing RoastMyPost: LLMs Eval Blog Posts and More

There's more going on, though that doesn't mean it will be necessarily better for you than your system.

There are some "custom" evaluators, that are basically just what you're describing, but with specific prompts. Though in these cases, note that there's extra functionality for users to re-run evaluators, see the histories of the runs, and see the specific agent's evals of many different documents.

The "system" evaluators typically have more specific code. They have short readmes you can see more on their pages:
https://www.roastmypost.org/evaluators/system-fact-checker
https://www.roastmypost.org/evaluators/system-fallacy-check

https://www.roastmypost.org/evaluators/system-forecast-checker

https://www.roastmypost.org/evaluators/system-link-verifier

https://www.roastmypost.org/evaluators/system-math-checker

https://www.roastmypost.org/evaluators/system-spelling-grammar

Some of this is just the tool splitting up a post into chunks, then doing analysis on each chunk. Some are more different. The link verifier works without any AI.

One limitation of... (read more)

Shallow review of technical AI safety, 2025

technicalities

technicalities, Tomáš Gavenčiak, Stephen McAleese, peligrietzer, Stag, jordine, ozziegooen, Violet Hour, ramennaut

2mo

Website version · Gestalt · Repo and data

Change in 18 latent capabilities between GPT-3 and o1, from Zhou et al (2025)

This is the third annual review of what’s going on in technical AI safety. You could stop reading here and instead explore the data on the shallow review website.

It’s shallow in the sense that 1) we are not specialists in almost any of it and that 2) we only spent about two hours on each entry. Still, among other things, we processed every arXiv paper on alignment, all Alignment Forum posts, as well as a year’s worth of Twitter.

It is substantially a list of lists structuring 800 links. The point is to produce stylised... (read 24743 more words →)

177

•••

Announcing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen

2mo

Today we're releasing RoastMyPost, a new experimental application for blog post evaluation using LLMs. Try it Here.

TLDR

RoastMyPost is a new QURI application that uses LLMs and code to evaluate blog posts and research documents.
It uses a variety of LLM evaluators. Most are narrow checks: Fact Check, Spell Check, Fallacy Check, Math Check, Link Check, Forecast Check, and others.
Optimized for EA & Rationalist content with direct import from EA Forum and LessWrong URLs. Other links use standard web fetching.
Works best for 200 - ~10,000 word documents with factual assertions and simple formatting. It can also do basic reviewing of Squiggle models. Longer documents and documents in LaTeX will experience slowdowns and errors.
Open source,

... (read 1394 more words →)

110

(Quick Thought)

Perhaps the goal for existing work targeting AI safety is less to ensure that AI safety happens, and more to make sure that we make AI systems that are strictly[1] better than the current researchers at figuring out what to do about AI safety.

I'm unsure how hard AI safety is. But I consider it fairly likely that mid-term (maybe 50% of the way to TAI, in years) safe AI systems are likely to outperform humans on AI safety strategy and the large majority of the research work.

If humans can successfully bootstrap more capable infrastructure than us, then our (humans) main work is done (though there could still be other work we... (read more)

Information-Dense Conference Badges

ozziegooen

8mo

See previous discussion here.

I find a lot of professional events fairly soul-crushing and have been thinking about why.

I dislike small talk. Recently I attended Manifest, and noticed that it could easily take 10 minutes of conversation to learn the very basics about a person. There were hundreds of people at the conference, so meeting people felt expensive and haphazard.

My regular time is costly, and conference time should be even more expensive. It seems reasonable to aim for $50 to $300 of value per hour. So if I spend 10 minutes with someone only to find out we don't have much in common, that can easily be $20 to $100 lost (total, between... (read 963 more words →)

Nuanced Models for the Influence of Information

ozziegooen

10mo

A common fallacy I see regarding information sources is the assumption that they either have zero impact or completely determine critical decisions.

"Our World In Data isn't directly cited by the US President, therefore it's worthless."
"Prediction markets haven't been adopted by top elected officials, so they contribute zero value."
"Prediction markets are already more competent than many intellectuals, and people don't listen to them. Therefore, there's no value in them being more competent. It's a bottleneck issue, and all that matters now is marketing."
"LLMs can't match the reasoning of elites, therefore they offer nothing."
"The moment LLM forecasters outperform human forecasting, in certain experiments, LLMs will completely take over."

This binary thinking misrepresents how information actually... (read 235 more words →)

If you've ever written or interacted with Squiggle code before, we at QURI would really appreciate it if you could fill out our Squiggle Survey!

https://docs.google.com/forms/d/e/1FAIpQLSfSnuKoUUQm4j3HEoqPmTYiWby9To8XXN5pDLlr95AiKa2srg/viewform

We don't have many ways to gauge or evaluate how people interact with our tools. Responses here will go a long way to deciding on our future plans.

Also, if we get enough responses, we'd like to make a public post about ways that people are (and aren't) using Squiggle.

Instead of "Goodharting", I like the potential names "Positive Alignment" and "Negative Alignment."

"Positive Alignment" means that the motivated party changes their actions in ways the incentive creator likes. "Negative Alignment" means the opposite.

Whenever there are incentives offered to certain people/agents, there are likely to be cases of both Positive Alignment and Negative Alignment. The net effect will likely be either positive or negative.

"Goodharting" is fairly vague and typically just refers to just the "Negative Alignment" portion.

I'd expect this to make some discussion clearer.
"Will this new incentive be goodharted?" -> "Will this incentive lead to Net-Negative Alignment?"

Other Name Options

Claude 3.7 recommended other naming ideas like:

Intentional vs Perverse Responses
Convergent vs Divergent Optimization
True-Goal vs Proxy-Goal Alignment
Productive vs Counterproductive Compliance

If we could have LLM agents that could inspect other software applications (including LLM agents) and make strong claims about them, that could open up a bunch of neat possibilities.

There could be assurances that apps won't share/store information.
There could be assurances that apps won't be controlled by any actor.
There could be assurances that apps can't be changed in certain ways (eventually).

I assume that all of this should provide most of the benefits people ascribe to blockchain benefits, but without the costs of being on the blockchain.

Some neat options from this:

Companies could request that LLM agents they trust inspect the code of SaaS providers, before doing business with them. This would be ongoing.
These

... (read more)

A potential future, focused on the epistemic considerations:

It's 2028.

MAGA types typically use DeepReasoning-MAGA. The far left typically uses DeepReasoning-JUSTICE. People in the middle often use DeepReasoning-INTELLECT, which has the biases of a somewhat middle-of-the-road voter.

Some niche technical academics (the same ones who currently favor Bayesian statistics) and hedge funds use DeepReasoning-UNBIASED, or DRU for short. DRU is known to have higher accuracy than the other models, but gets a lot of public hate for having controversial viewpoints. DRU is known to be fairly off-putting to chat with and doesn't get much promotion.

Bain and McKinsey both have their own offerings, called DR-Bain and DR-McKinsey, respectively. These are a bit like DeepReasoning-INTELLECT, but are... (read 432 more words →)

Quick list of some ideas I'm excited about, broadly around epistemics/strategy/AI.

1. I think AI auditors / overseers of critical organizations (AI efforts, policy groups, company management) are really great and perhaps crucial to get right, but would be difficult to do well.

2. AI strategists/tools telling/helping us broadly what to do about AI safety seems pretty safe.

3. In terms of commercial products, there’s been some neat/scary military companies in the last few years (Palantir, Anduril). I’d be really interested if there could be some companies to automate core parts of the non-military government. I imagine there are some parts of the government that are particularly tractable/influenceable/tractable. For example, just making great decisions on... (read more)

6 (Potential) Misconceptions about AI Intellectuals

ozziegooen

One of my key thoughts is that the bar of “human intellectuals” is just really low. I think that intellectual work is useful and that these intellectuals on the whole do produce some value, but I also think we can do much better.

Epistemic Status
A collection of thoughts I've had over the last few years, lightly edited using Claude. I think we're at the point in this discussion where we need to get the basic shape of the conversation right. Later we can bring out more hard data.

Summary

While artificial intelligence has made impressive strides in specialized domains like coding, art, and medicine, I think its potential to automate high-level strategic thinking has been... (read 3423 more words →)

It's arguably difficult to prove that AIs can be as good or better at moral reasoning than humans.

A lot of the challenge is that there's no clear standard for moral reasoning. Honestly, I'd guess that a big part of this is that humans are generally quite bad at it, and generally highly overconfident in their own moral intuitions.

But one clearer measure is if AIs can predict human's moral judgements. Very arguably, if an AI system can predict all the moral beliefs that a human would have after being exposed to different information, then the AI must be capable of doing as good a job at moral reasoning.

There is a very different question... (read more)

In terms of proposing and discussing AI Alignment strategies, I feel like a few individuals have been dominating the LessWrong conversation recently.

I've seen a whole lot from John Wentworth and the Redwood team.

After that, it seems to get messier.

There are several individuals or small groups with their own very unique takes. Matthew Barnett, Davidad, Jesse Hoogland, etc. I think these groups often have very singular visions that they work on, that few others have much buy-in with.

Groups like the Deepmind and Anthropic safety teams seem hesitant to write much about or discuss big-picture strategy. My impression is that specific researchers are working typically working on fairly narrow agendas, and that the leaders... (read more)

A bunch of people in the AI safety landscape seem to argue "we need to stop AI progress, so that we can make progress on AI safety first."

One flip side to this is that I think it's incredibly easy for people to waste a ton of resources on "AI safety" at this point.

I'm not sure how much I trust most technical AI safety researchers to make important progress on AI safety now. And I trust most institutions a lot less.

I'd naively expect if any major country would throw $100 Billion on it today, the results would be highly underwhelming. I rarely trust these governments to make progress on concrete technologies with clear... (read more)

There have been a few takes so far of humans gradually losing control to AIs - not through specific systems going clearly wrong, but rather by a long-term process of increasing complexity and incentives.

This sometimes gets classified as "systematic" failures - in comparison to "misuse" and "misalignment."

There was "What Failure Looks Like", and more recently, this piece on "Gradual Disempowerment."

To me, these pieces come across as highly hand-wavy, speculative, and questionable.

I get the impression that a lot of people have strong low-level assumptions here that a world with lots of strong AIs must go haywire. But I don't see clear steps to get there or a clear model of what the critical... (read more)

$300 Fermi Model Competition

ozziegooen

Summary

Task: Make an interesting and informative Fermi estimate
Prize: $300 for the top entry
Deadline: February 16th, 2025
Results Announcement: By March 1st, 2025
Judges: Claude 3.5 Sonnet, the QURI team

Motivation

LLMs have recently made it significantly easier to make Fermi estimates. You can chat with most LLMs directly, or you can use custom tools like Squiggle AI. And yet, overall, few people have taken much advantage of this.

We at QURI are launching a competition to encourage exploration.

What We’re Looking For

Our goal is to discover creative ways to use AI for Fermi estimation. We're more excited about novel approaches than exhaustively researched calculations. Rather than spending hours gathering statistics or building complex spreadsheets, we encourage you to:

Let AI do most

... (read 342 more words →)

AI for Resolving Forecasting Questions: An Early Exploration

ozziegooen

Thanks to Slava Matyuhin for comments

Summary

AIs can be used to resolve forecasting questions on platforms like Manifold and Metaculus.
AI question resolution, in theory, can be far more predictable, accessible, and inexpensive to human resolution.
Current AI tools (combinations of LLM calls and software) are mediocre at judging subjective or speculative questions. We expect them to improve.
AI tools are changing rapidly. If we want to organize forecasting questions to be resolved using an AI tool in 10 years, instead of selecting a specific tool, we might want to select a protocol for choosing the best tool at resolution time. We call this an "Epistemic Selection Protocol".
If we have a strong "Epistemic Selection Protocol" that

... (read 2553 more words →)

Introducing Squiggle AI

ozziegooen

We’re releasing Squiggle AI, a tool that generates probabilistic models using the Squiggle language. This can provide early cost-effectiveness models and other kinds of probabilistic programs.

No prior Squiggle knowledge is required to use Squiggle AI. Simply ask for whatever you want to estimate, and the results should be fairly understandable. The Squiggle programming language acts as an adjustable backend, but isn’t mandatory to learn.

Beyond being directly useful, we’re interested in Squiggle AI as an experiment in epistemic reasoning with LLMs. We hope it will help highlight potential strengths, weaknesses, and directions for the field.

Screenshots

The “Playground” view after it finishes a successful workflow. Form on the left, code in the middle, code output

... (read 2166 more words →)

Enhancing Mathematical Modeling with LLMs: Goals, Challenges, and Evaluations

ozziegooen

Introduction

Motivation

At QURI, we've been investing in developing and analyzing mathematical models, recently focusing on Squiggle AI, a system that leverages LLMs to generate models using the Squiggle language. We believe LLMs hold substantial potential for improving reasoning and are keen to explore how best to integrate them with mathematical modeling.

While recent attempts to automate narrow forecasts using LLMs have shown promise, mathematical models can offer far greater expressiveness and power. We're particularly interested in LLMs' ability to generate complex forecasts and explanations through formal modeling. However, the lack of clear evaluations or benchmarks for these models poses a challenge. This essay aims to dissect the key aspects of mathematical models and suggest ways... (read 4415 more words →)

What's Going on With OpenAI's Messaging?

Shallow review of technical AI safety, 2025

Working in Virtual Reality: A Review

Announcing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen

Opinion Fuzzing: A Proposal for Reducing & Exploring Variance in LLM Judgments Via Sampling

Shallow review of technical AI safety, 2025

Announcing RoastMyPost: LLMs Eval Blog Posts and More

Information-Dense Conference Badges

Nuanced Models for the Influence of Information

6 (Potential) Misconceptions about AI Intellectuals

$300 Fermi Model Competition

Beyond Questions & Answers

Squiggle

Prediction-Driven Collaborative Reasoning Systems

ozziegooen

What's Going on With OpenAI's Messaging?

Shallow review of technical AI safety, 2025

Working in Virtual Reality: A Review

Announcing RoastMyPost: LLMs Eval Blog Posts and More

ozziegooen

Opinion Fuzzing: A Proposal for Reducing & Exploring Variance in LLM Judgments Via Sampling

Shallow review of technical AI safety, 2025

Announcing RoastMyPost: LLMs Eval Blog Posts and More

Information-Dense Conference Badges

Nuanced Models for the Influence of Information

6 (Potential) Misconceptions about AI Intellectuals

$300 Fermi Model Competition

Beyond Questions & Answers

Squiggle

Prediction-Driven Collaborative Reasoning Systems

Introduction

TLDR

Summary

Summary

Motivation

What We’re Looking For

Summary

Screenshots

Introduction

Motivation