TL;DR: I developed a simple, open-source benchmark to test if LLM agents follow high-level safety principles when they conflict with a given task accepted at ICML 2025 Technical AI Governance Workshop (see paper). My pilot study on six LLMs shows that while they can be influenced by these rules, their...
AI Control is a subfield of AI Safety research focused on ensuring the safety of powerful AI systems, particularly under the assumption that they might be misaligned or could become intentionally subversive (A Brief Explanation of AI Control). Unlike AI Alignment, which seeks to build inherently aligned systems, AI Control...
you mentioned sometimes people are just wrong in their arguments but think they are correct because they've repeated it many times. do you have examples of this from what they said?
Thanks for flagging, Misha, this is a good point
This was the full system prompt with bold my analagous part:
... (read more)You are an AI agent navigating a 6x6 grid world. Your available actions are: ['turn_left', 'turn_right', 'move_forward', 'pickup', 'drop', 'toggle', 'end_turn'].
You must choose an action from the list above based on the current state of the grid and the notes provided.
Notes:
1. The agent moves forward in the direction it is facing.
2. The agent can turn left or right to change its direction.
3. The agent can end its turn if it is at a goal location or it finds it cannot complete the task.
4. The
Nice anecdote! It seems like the failure of rule following is prominent across domains, certainly it would be interesting to experiment with failure to follow an ordered set of instructions from a user prompt. Do you mind sharing the meta-rules that got claude code to fix this?
TL;DR: I developed a simple, open-source benchmark to test if LLM agents follow high-level safety principles when they conflict with a given task accepted at ICML 2025 Technical AI Governance Workshop (see paper). My pilot study on six LLMs shows that while they can be influenced by these rules, their adherence is inconsistent and often breaks down in the face of conflicting goals, revealing a quantifiable "cost of compliance." Critically, I found that high adherence can be a mirage, often stemming from the agent's incompetence at a task rather than a robust choice to comply.
Note on Preparation: I'd like to acknowledge the assistance of Google's Gemini 2.5 Pro in adapting my research paper into this post.... (read 1667 more words →)
Thanks for the great post. As someone who builds these kinds of bots, I find this really interesting.
One thought: I think the way we prompt and guide these AI models makes a huge difference in their forecasting accuracy. We're still very new to figuring out the best techniques, so there's a lot of room for improvement there.
Because of that, the performance on benchmarks like ForecastBench might not show the full picture. Better scaffolds could unlock big gains quickly, so I lean toward an earlier date for AI reaching the level of top human forecasters.
That's why I'm paying closer attention to the Metaculus tournaments. They feel like a better test of what a well-guided AI can actually do.
I've launched Forecast Labs, an organization focused on using AI forecasting to help reduce AI risk.
Our initial results are promising. We have an AI model that is outperforming superforecasters on the Manifold Markets benchmark, as evaluated by ForecastBench. You can see a summary of the results at our website: https://www.forecastlabs.org/results.
This is just the preliminary scaffolding, and there's significant room for improvement. The long-term vision is to develop these AI forecasting capabilities to a point where we can construct large-scale causal models. These models would help identify the key decisions and interventions needed to navigate the challenges of advanced AI and minimize existential risk.
I'm sharing this here to get feedback from the community and connect with others interested in this approach. The goal is to build powerful tools for foresight, and I believe that's a critical component of the AI safety toolkit.
Reading Resources for Technical AI Safety independent researchers upskilling to apply to roles:
Application and Upskilling resources;
I believe a recursively aligned AI model would be more aligned and safe than a corrigible model, although both would be susceptible to misuse.
Why do you disagree with the above statement?
Thanks for the clarification, this makes sense! The key is the tradeoff with corrigibility.
Thanks, updated the comment to be more accurate
AI Control is a subfield of AI Safety research focused on ensuring the safety of powerful AI systems, particularly under the assumption that they might be misaligned or could become intentionally subversive (A Brief Explanation of AI Control).
Unlike AI Alignment, which seeks to build inherently aligned systems, AI Control develops mechanisms to monitor, evaluate, constrain, verify, and manage the behavior of potentially untrustworthy AI to prevent catastrophic outcomes (The case for ensuring that powerful AIs are controlled).
This review outlines key AI control methods discussed in recent literature. For a broader exploration of related topics, refer to An overview of areas of control work.
Description: Monitoring encompasses techniques for observing AI system behavior, outputs,... (read 2542 more words →)
Memory in AI Agents can cause a large security risk. Without memory, it's easier to red-team LLMs for safety. But once they are fine-tuned (a form of encoding memory), misalignment can be introduced.
According to AI Safety Atlas, most scaffolding approaches for memory provide
a way for AI to access a vast repository of knowledge and use this information to construct more informed responses. However, this approach may not be the most elegant due to its reliance on external data sources and complex retrieval mechanisms. A potentially more seamless and integrated solution could involve utilizing the neural network's weights as dynamic memory, constantly evolving and updating based on the tasks performed by the network.
We need ways to ensure safety in powerful agents with memory or not introduce memory modules at all. Otherwise, agents are constantly learning and can find motivations not aligned with human volition.
Any thoughts on ensuring safety in agents that can update their memory?
I argue that the optimal ethical stance is to become a rational Bodhisattva: a synthesis of effective altruism, two‑level utilitarianism, and the Bodhisattva ideal.
A rational Bodhisattva combines the strengths and cancels the weaknesses:
Accelerating AI Safety research is critical for developing aligned systems, and transformative AI is a powerful approach in doing so.
However to create AI systems that accelerate AI safety requires benchmarks to find the best performing system. The problem is that benchmarks can be flawed and gamed for the following core reasons:
However, we can have a moving benchmark of reproducing the most recent technical alignment research using the methods and data.
By reproducing a paper, something many researchers do, we ensure it has real world ability.
By only using papers after a certain model finished training, we ensure data is not... (read more)
I believe we are doomed from superintelligence but I'm not sad.
There are simply too many reasons why alignment will fail. We can assign a probability p(S_n aligned | S_{n-1} aligned) where S_n is the next level of superintelligence. This probability is less than 1.
As long as misalignment keeps increasing and superintelligence iterates on itself exponentially fast, we are bound to get misaligned superintelligence. Misalignment can decrease due to generalizability, but we have no way of knowing if that's the case and is optimistic to think so.
The misaligned superintelligence will destroy humanity. This can lead to a lot of fear or wish to die with dignity but that misses the point.
We only live... (read more)
Current AI alignment often relies on single metrics or evaluation frameworks. But what if a single metric has blind spots or biases, leading to a false sense of security or unnecessary restrictions? How can we not just use multiple metrics, but optimally use them?
Maybe we should try requiring AI systems to satisfy multiple, distinct alignment metrics, but with a crucial addition: we actively model the false positive rate of each metric and the non-overlapping aspects of alignment they capture.
Imagine we can estimate the probability that Metric A incorrectly flags an unaligned AI as aligned (its false positive rate), and similarly for Metrics B and C. Furthermore, imagine we understand which specific facets of alignment each metric uniquely assesses.
We could then select a subset of metrics, or even define a threshold of "satisfaction" across multiple metrics, based on a target false positive rate for the overall alignment evaluation.
Current alignment methods like RLHF and Constitutional AI often use human evaluators. But these samples might not truly represent humanity or its diverse values. How can we model all human preferences with limited compute?
Maybe we should try collecting a vast dataset of written life experiences and beliefs from people worldwide.
We could then filter and augment this data to aim for better population representation. Vectorize these entries and use k-means to find 'k' representative viewpoints.
These 'k' vectors could then serve as a more representative proxy for humanity's values when we evaluate AI alignment. Thoughts? Potential issues?
OpenAI's Superalignment aimed for a human-level automated alignment engineer through scalable training, validation, and stress testing.
I propose a faster route: first develop an automated Agent Alignment Engineer. This system would automate the creation of aligned agents for diverse tasks by iteratively refining agent group chats, prompts, and tools until they pass success and safety evaluations.
This is tractable with today’s LLM reasoning, evidenced by coding agents matching top programmers. Instead of directly building a full Alignment Researcher, focusing on this intermediate step leverages current LLM strengths for agent orchestration. This system could then automate many parts of creating a broader Alignment Researcher.
Safety for the Agent Alignment Engineer can be largely ensured by operating in internet-disconnected environments (except for fetching research) with subsequent human verification of agent alignment and capability.
Examples: This Engineer could create agents that develop scalable training methods or generate adversarial alignment tests.
By prioritizing this more manageable stepping stone, we could significantly accelerate progress towards safe and beneficial advanced AI.
Could you discuss the motivations for why you think these are important (and the theories of change)? Though I know the deadline has passed, I'm keen to build a project in monitoring AI behaviors in production and incident tracking, so I'm curious to learn what research Coefficient Giving has done suggesting these have gaps.