Independent AI alignment researcher. I hold an MSc equivalent degree in psychology from the University of Tartu. I work as an AI software architect specialising in combinatorial optimisation, machine learning, graph search, natural language processing, and data compression.

I have been both researching and professionally working on multi-objective value problems...

This conceptual overview post is intended to explain what I mean by the principles of "homeostasis", "diminishing returns", and "balancing" - how these ideas differ, complement, and interact with each other. Alongside, there is also an overview of my research agenda.

What am I trying to promote, in simple words

I want to build and promote AI systems that are trained to understand and follow two fundamental principles from biology and economics:

Moderation - Enables the agents to understand the concept of “enough” versus “too much”. The agents would understand that too much of a good thing would be harmful even for the very objective that was maximised for, and they would actively avoid such

... (read 2203 more words →)

Fairly-direct alignment via changing training to reflect actual human reward.

Unless I misunderstand the idea of the highlighted sentence then I believe the following post is also motivated by very much same themes:
Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and Open Challenges.

It is essentially about utility / reward functions in the brain and how naive unbounded maximisation is partially alien to biological / human needs. Many or even almost all biological needs require the target objectives to be in an optimal range - both too little and too much must be actively avoided.

If AI training (and model default assumptions / mathematics) do not reflect or optimally support these considerations then it is likely unaligned from the start.

There is still an important place for unbounded objectives, but it seems unboundedness is appropriate primarily for instrumental objectives.

Hello! Posted a new document on brainstorming for methodology of further research on the current runaway LLM-s findings:

Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs

I am hoping that this post will serve also as a generally interesting brainstorming collection and discussion ground of black-box LLM interpretability methodology as well as failure mitigation ideas.

Hope you find it relevant and interesting!

Working document – feedback very welcome. Ideas in the main body are those I currently see as highest-leverage; numerous items under Appendix are more tentative and would benefit from critique as well as additions.

I am hoping that this post will serve also as a generally interesting brainstorming collection and discussion ground of black-box LLM interpretability methodology as well as failure mitigation ideas.

I would like to thank my collaborators: Sruthi Susan Kuriakose, Aintelope members (Andre Addis, Angie Normandale, Gunnar Zarncke, Joel Pyykkö, Rasmus Herlo), Kabir Kumar @ AI-Plans and Stijn Servaes @ AE Studio for stimulating discussions and shared links leading to these ideas. All possible mistakes are mine.

TL;DR – A Shortened Version

This chapter is... (read 1989 more words →)

Sharing a couple of new links related to your question about more details. Hope you find them interesting!

Presentation at MAISU unconference 2025:

Link to slides: https://bit.ly/beab-llm
Session recording at YouTube: https://bit.ly/beab-llm-rec

Link to annotated data files

https://bit.ly/beab-llm-data

(Each file has multiple sheets. Only trials with failures are provided.)

In this project it seems looking at the annotated logs of individual runs is much more interesting than looking at the aggregated plots over many runs. This is because:

The failures start at different timesteps in each run, aggregation obfuscates that information.
The single objective optimisation could focus on either objective A or B during different experiment trials, but during aggregation both objectives would get equalised again.
The self-imitation patterns cannot be preserved

... (read more)

Thank you for asking! I am encouraged to hear you find it important :) It is under Links section. Code, system prompts, output data files, plots, and a more detailed report can be found here: https://github.com/biological-alignment-benchmarks/bioblue

Please let me know if you have any further questions! The GitHub readme document and PDF report contain a lot of information and maybe the structure could be improved in the future.

The Python code files for the four experiments are runnable independently from each other and are relatively short, about 250 - 300 lines each. To see the nuances of the results you might want to take a closer look at the output data files with your own eye.

This enables you to see that the models fail at different points in time, but the sequence of actions after failure point is usually typical. In the coming days, I will create one more output folder with annotated log files so the failure modes can be found more easily.

I renamed the phenomenon to "runaway optimiser". I hope this label illustrates the inappropriately unbounded and single-minded nature of the failure modes we observed. How does that sound to you, does that capture the essence of the phenomena described in the post?

Thank you for pointing that out! I agree, there are couple of nuances. Our perspective can be treated as a generalisation of the original utility monster scenario. Although I consider it to be not first such generalisation - think of the examples in Bostrom's book.

1) In our case, the dilemma is not "agent versus others", but instead "one objective versus other objectives". One objective seems to get more internal/subjective utility from consumption than another objective. Thus the agent focuses on a single objective only.
2) Consideration of homeostatic objectives introduces a new aspect to the utility monster problem - the behaviour of the original utility monster looks unaligned to begin with, not just dominating.... (read more)

By Roland Pihlakas, Sruthi Susan Kuriakose, Shruti Datta Gupta

This research is now available as an Arxiv preprint at https://arxiv.org/abs/2509.02655 . The paper includes tables with select snippets of failure mode examples.

Summary and Key Takeaways

Relatively many past AI safety discussions have centered around the dangers of unbounded utility maximisation by RL agents, illustrated by scenarios like the "paperclip maximiser" or by specification gaming in general. Unbounded maximisation is problematic for many reasons. We wanted to verify whether these RL runaway optimisation problems are still relevant with LLMs as well.

Turns out, strangely, this is indeed clearly the case. The problem is not that the LLMs just lose context or become incoherent. The problem is that in various... (read 4290 more words →)

I agree, sounds plausible that this could happen. Likewise as we humans may build a strongly optimising agent because we are lazy and want to use simpler forms of maths. The tiling agents problem is definitely important.

That being said, agents properly understanding and modelling homeostasis is among the required properties (thus essential). It is not meant to be sufficient one. There may be no single sufficient property that solves everything, therefore there is no competition between different required properties. Required properties are conjunctive, they are all needed. My intuition is that homeostasis is one such property. If we neglect homeostasis then we are likely in trouble regardless of advances in other properties.

If... (read 522 more words →)

I notice that there has been very little discussion on why and how considering homeostasis is significant, even essential for AI alignment and safety. The current post aims to contribute to or assamending that situation. In this post I will treat alignment and safety as explicitly separate subjects, which both benefit from homeostatic approaches.

This text is a distillation and reorganisation of three of my older blog posts at Medium:

I will probably share more such distillations or weaves... (read 4344 more words →)

Thank you for your question!

I agree that the simulations need to have sufficient complexity. Indeed, that was one of main motivations I became interested in creating multi-objective benchmarks in the past. Various AI safety toy problems seemed to me so much simplified that they lacked essential objectives and other decisive nuances. This motivation is still very much one of my main driving motivations.

That being said, complexity has also downsides:
1) The complexity introduces confounding factors. When a model fails such a benchmark, it is not clear whether it was because it did not have required perceptual capabilities (so it is a capabilities problem), or it is using a model/framework that is unsuitable for... (read more)

This is an AI Safety Camp 10 project that I will be leading. With this post, I am looking for external collaborators, ideas, questions, resource suggestions, feedback, and other thoughts.

Summary

Based on various sources of anthropological research, I have compiled a preliminary list of universal (cross-cultural) human values. It seems to me that various of these universal values resonate with concepts from AI safety, but use different keywords. It might be useful to map these universal values to more concrete definitions using concepts from AI safety.

One notable detail in this research is that in case of AI and human cooperation, the values are not symmetric as they would be in case of human-human cooperation.... (read 3290 more words →)

I think your own message is also too extreme to be rational. So it seems to me that you are fighting fire with a fire. Yes, Remmelt has some extreme expressions, but you definitely have extreme expressions here too, while having even weaker arguments.

Could we find a golden middle road, a common ground, please? With more reflective thinking and with less focus on right and wrong?

I agree that Remmelt can improve the message. And I believe he will do that.

I may not agree that we are going to die with 99% probability. At the same time I find that his current directions are definitely worthwhile of exploring.

I also definitely respect Paul. But mentioning... (read more)

The following is meant as a question to find out, not a statement of belief.

Nobody seems to have mentioned the possibility that initially they did not intend to fire Sam, but just to warn him or to give him a choice to restrain himself. Yet possibly he himself escalated it to firing or chose firing instead of complying with the restraint. He might have done that just in order to have all the consequences that have now taken place, giving him more power.

For example, people in power positions may escalate disagreements, because that is a territory they are more experienced with as compared to their opponents.

Background: A multi-objective decision-making AI

Previously I've proposed balancing multiple objectives via multi-objective RL as a method to achieve AI Alignment. If we want an AI to achieve goals including maximizing human preferences, or human values, but also maximizing corrigibility, and interpretability, and so on--perhaps the key is to simply build a system with a goal to maximize all those things. With Peter Vamplew and other co-authors, I have previously argued multi-objective RL is superior for human-aligned decision-making relative to scalar [single-objective] RL.

This post describes, if one was to try and implement a multi-objective reinforcement learning agent that optimized multiple objectives, what those objectives might look like, concretely. I've attempted to describe some specific problems... (read 2228 more words →)

By Ben Smith, Roland Pihlakas, and Robert Klassert

Thanks to Linda Linsefors, Alex Turner, Richard Ngo, Peter Vamplew, JJ Hepburn, Tan Zhi-Xuan, Remmelt Ellen, Kaj Sotala, Koen Holtman, and Søren Elverlin for their time and kind remarks in reviewing this essay. Thanks to the organisers of the AI Safety Camp for incubating this project from its inception and for connecting our team.

For the last 9 months, we have been investigating the case for a multi-objective approach to reinforcement learning in AI Safety. Based on our work so far, we’re moderately convinced that multi-objective reinforcement learning should be explored as a useful way to help us understand ways in which we can achieve safe... (read 2992 more words →)

LESSWRONG
LW

LESSWRONG
LW

Roland Pihlakas

Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and open challenges.

Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format (BioBlue)

A brief review of the reasons multi-objective RL could be important in AI Safety Research

Building AI safety benchmark environments on themes of universal human values

Roland Pihlakas

Research agenda for training aligned AIs using concave utility functions following the principles of homeostasis and diminishing returns

Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs

Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format (BioBlue)

Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and open challenges.

Building AI safety benchmark environments on themes of universal human values

Sets of objectives for a multi-objective RL agent to optimize

A brief review of the reasons multi-objective RL could be important in AI Safety Research

Roland Pihlakas

Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and open challenges.

Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format (BioBlue)

A brief review of the reasons multi-objective RL could be important in AI Safety Research

Building AI safety benchmark environments on themes of universal human values

Roland Pihlakas

Research agenda for training aligned AIs using concave utility functions following the principles of homeostasis and diminishing returns

Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs

Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format (BioBlue)

Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and open challenges.

Building AI safety benchmark environments on themes of universal human values

Sets of objectives for a multi-objective RL agent to optimize

A brief review of the reasons multi-objective RL could be important in AI Safety Research

What am I trying to promote, in simple words

TL;DR – A Shortened Version

Summary and Key Takeaways

Summary

Background: A multi-objective decision-making AI