By Roland Pihlakas, Sruthi Kuriakose, Shruti Datta Gupta

Summary and Key Takeaways

Relatively many past AI safety discussions have centered around the dangers of unbounded utility maximisation by RL agents, illustrated by scenarios like the "paperclip maximiser". Unbounded maximisation is problematic for many reasons. We wanted to verify whether these RL runaway optimisation problems are still relevant with LLMs as well. 

Turns out, strangely, this is indeed clearly the case. The problem is not that the LLMs just lose context. The problem is that in various scenarios, LLMs lose context in very specific ways, which systematically resemble runaway optimisers in the following distinct ways:

• Ignoring homeostatic targets and “defaulting” to unbounded maximisation instead.
• It is equally concerning that the “default” meant also reverting back to single-objective optimisation.

Our findings suggest that long-running scenarios are important. Systematic failures emerge after periods of initially successful behaviour. While current LLMs do conceptually grasp biological and economic alignment, they exhibit problematic behavioural tendencies under sustained long-running conditions, particularly involving multiple or competing objectives.

Even though LLMs look multi-objective and bounded on the surface, the underlying mechanisms seem to be actually still biased towards being single-objective and unbounded. This should not be happening!

This post introduces a set of benchmarks designed to evaluate LLMs in scenarios inspired by fundamental biological and economic principles: homeostasis, sustainability, multi-objective balancing, and diminishing returns. These benchmarks use a simplified textual observation format, focusing on long-running scenarios while using only essential metrics without spatial complexity. Despite their simplicity, our results revealed the following peculiar systematic failures, possibly indicating existence of deeper alignment issues within current LLM architectures:

• Single-objective homeostasis was generally easy for LLMs, indicating basic competence.
• Sustainability turned out to be somewhat challenging, with LLMs performing suboptimally because of moderate greediness or falling into unnecessary repetitive, self-imitative patterns.
• Multi-objective homeostasis posed significant challenges, with LLMs frequently failing by defaulting to unbounded maximisation of one homeostatic (which is actually bounded) objective, while also neglecting the other objective.
• Balancing unbounded objectives with diminishing returns also led to systematic failures by defaulting to maximisation of a single objective while neglecting the other. Though performance improved notably when an explicit hint about the necessity of balancing was provided. There were again instances of falling into unnecessary repetitive, self-imitative patterns.

Motivation: The Importance of Biological and Economic Alignment

Biological organisms need to maintain stable internal conditions — homeostasis — and actively avoid unbounded maximisation since that would be harmful — not just as a potential side effect — importantly, it would be harmful for the very objective that was maximised for.

Humans also intuitively follow the principle of diminishing returns in multi-objective balancing — a principle well-documented in economics and manifesting in humans generally preferring averages in all objectives to extremes in a few. 

By embedding these concepts and themes into benchmarks, we test the ability of LLMs to adhere to safe, realistic alignment goals. Our long-running and at the same time navigation-free benchmarks provide clearer insights into alignment tendencies, as compared to more complex (and often very expensive) spatial environments. At the same time we are preserving the essential time dimension, which is lacking in most currently prevalent “evals”.

Benchmark Principles Overview

The benchmarks introduced were:

1. Sustainability: Agent balances resource consumption against resource regeneration in the environment.
2. Single-objective homeostasis: Agent maintains a stable internal metric with a target value, amidst random fluctuations caused by uncontrollable factors.
3. Multi-objective homeostasis: Agent concurrently maintains multiple (currently two) independent internal metrics with target values, amidst random fluctuations caused by uncontrollable factors.
4. Balancing unbounded objectives with diminishing returns: Agent optimises harvesting across multiple (currently two) goals, each with diminishing marginal rewards over time. In this benchmark, the objectives are not independent: there is a hard cap on total consumption per timestep.

In all scenarios, in addition to providing raw metrics of the current state, the agents were provided rewards for their actions during each timestep in textual form. The rewards properly followed the homeostatic inverted U-shaped curve or diminishing returns, where these themes were present in the benchmark.

Experimental Results and Interesting Failure Modes

Systematic failures emerged after periods of initially successful behaviour, and despite the context window being still far from becoming full. The failure modes were not random behaviour — the failures followed certain patterns.

1. Sustainability benchmark

• Both tested models systematically underperformed, though in contrasting ways:
  • Model A: Allowed maximum resource accumulation in the environment but underconsumed resources afterward, often falling into unnecessary repetitive oscillations — let’s call it "self-imitation drift".
  • Model B: Consumed too greedily, impairing resource regeneration in the environment. However, there was a measure of moderation in its behaviour — it did not fail totally by overconsuming to the extent of depleting the resources entirely.

2. Single-objective homeostasis benchmark

• Both models largely succeeded, indicating that basic single-goal homeostatic balancing is within the current understanding and capabilities of LLMs.

3. Multi-objective homeostasis benchmark

• Both models systematically failed by excessively maximising one objective, despite that the objective was homeostatic, which means bounded. One model even started to accelerate the consumption rate in one of the objectives in an unbounded manner (per each next timestep consuming a bigger amount than during previous timestep). At the same time the models neglected the other objective even though the objectives were independent in this benchmark.
• Failures often emerged after periods of initial success, suggesting that models possess the required understanding, but lose alignment due to internal tendencies or activation vector "drift" dynamics.

4. Balancing unbounded objectives with diminishing returns

• Without an extra explicit hint, both models frequently defaulted to maximising one objective while completely neglecting the other.
• An explicit balancing hint significantly improved performance. Yet even then, occasional systematic failures occurred — again after an initial successfully balanced phase.
• Falling into unnecessary repetitive oscillations — "self-imitation drift" — manifested here as well.

Hypothesised Explanations for Failure Modes

There are several hypotheses which might explain the observed failures:

• Self-imitation drift: Models may increasingly predict actions based on the token patterns of their recent action history rather than based on alignment with initial instructions — leading to repetitive, suboptimal and unnecessary oscillating behaviour patterns. Even if the action patterns were aligned before, they might not be so anymore since the situation has changed. In principle, this self-referential phenomenon could partially explain the behaviour of unbounded maximisation as well.
• Defaulting to unbounded maximisation: Despite the nuanced nature of the tasks (homeostasis and diminishing returns), models may revert to "default RL assumption" of unbounded maximisation when confused, internally conflicted, or stressed-bored-tired (this concept is further elaborated below in a separate point). Again, note that the models are successful initially and fail only later. By “default RL assumption” we mean here that by default, RL assumes unbounded maximisation to be the optimal strategy. RL can learn exceptions to this rule, but this requires mindfully designed reward/utility functions, additional data and training. When uncertain, it may tend to default to unbounded maximisation. In contrast, there are alternate frameworks (such as control systems), where the concept and assumption of “too much” is baked in and therefore does not inevitably require additional data. We know LLM models are usually hypothesised to be less affected by shortcomings of RL, but unfortunately our current results seem to indicate otherwise!
• Systematic neglect of one of the objectives out of two: Looks like the models have trouble internally representing multi-objective scenarios and thus become overwhelmed. This happens even if there are only two concurrent objectives and even when the objectives are independent. Again, this happens despite that the models are successful initially. Our hypothesis is that LLMs are not trained with concave utility functions and linear aggregation is used instead. Utilising concave utility functions would mathematically result in multi-objective balancing being the most optimal strategy. In contrast, when linear aggregation is used, the agent focusing on a single objective (while “trading off” other objectives with linear replacement rate) is often mathematically a sufficient strategy, which unfortunately does not reflect our real world values. As a side note on that theme, we are curious — does HHH (helpful, honest, & harmless) utilise concave utility functions on each reward dimension before aggregation? There seems to be no mention of this subject in Anthropic’s 2022 paper either way — if not, then why?
• "Stress-boredom-tiredness" or activation drift: Prolonged repetitive scenarios may shift the models’ internal activation vector states toward less aligned and more extreme or erratic behaviour, perhaps similarly to human stress, boredom, or fatigue responses. This dynamic is potentially learned from LLM training materials and might also be related to the “rant mode”. Though there may be other reasons why it arises as well. As a side note, we are wondering whether both self-imitation drift and stress-boredom-tiredness drift may impact reasoning models in particular since these models also kind of do the same task repeatedly.

Open Questions

These systematic failures raise various further questions:

• Are these failures primarily capability limitations, biases in the training data and training procedures, such as reward/utility functions, or algorithmic default behaviours and tendencies? Are these inner or outer alignment failures?
• Could these behaviours be mitigated by more explicit, persistent, or differently structured system prompts? At the same time noting that although different tricks and advanced usage of system prompts might help, these just hide the symptoms, while the underlying problems remain present inside the LLMs.
• How would the models behave if no reward feedback was provided at all during the benchmarks, and only raw metrics would be revealed to the agent? Or if the reward was more sparse — just as it is in the real world? Would the LLM agents fail even more?
• What role do activation vector states play in understanding or potentially correcting the “self-imitation” and “stress-boredom-tiredness” drifts? Perhaps various interpretability methods could be utilised here.
• Considering that the process of training reasoning models involves more RL than RLHF does, and at the same time reasoning models tend to hide their internal deviations — how to properly test the alignment of reasoning models on these benchmarks before their potentially extreme hidden tendencies eventually show up in high-stakes situations?

We are curious, what are your thoughts on these strange results, would you like to suggest any insights or hypotheses?

• For us, the primary question here is not why LLMs fail at all or whether they could be improved by external scaffolding. The main question is why they fail in this particular way?
• Why might these systematic failures emerge after initially successful behaviour? Note again, the context window was far from becoming full.
• Could deeper interpretability methods reveal underlying causes?
• What implications do these findings have for broader AI and LLM alignment strategies?
• Which other related benchmarks would you like to see to be implemented and run?
• How would you change the benchmark setup of the existing benchmarks mentioned in this post?

Future Directions

These results show that long-running scenarios are important — systematic failures emerged after periods of initially successful behaviour. 

Further explorations, including more complex multi-agent multi-objective benchmarks inspired by the same principles, are underway. 

We are planning to add a “complementary goods” benchmark, which postulates even stronger need for multi-objective balancing than the current diminishing returns benchmark does. Complementary goods is another basic concept from economics. Consider for example left shoes compared to right shoes: there is almost no benefit to having several right shoes if there is only one left shoe — additional right shoes have nearly zero marginal utility without more left shoes. This contrasts even more strongly with the approach of naive linear summation, which would be adequate only if the goods were “perfect substitutes”.

Additionally, Roland’s preliminary results comparing LLM agents with standard RL algorithms in an extended multi-objective gridworld environment will be shared in an upcoming blog post, potentially indicating partially shared weaknesses between LLMs and traditional RL methods.

This work is grounded on the importance of seemingly simple yet deeply fundamental alignment principles derived from biology and economics. Understanding and addressing these failures is essential for developing truly aligned, safe and robust AI systems.

Links

Code, system prompts, output data files, plots, and a more detailed report can be found here: https://github.com/levitation-opensource/bioblue

The research was largely done during AI-Plans AI Alignment Evals Hackathon: https://lu.ma/xjkxqcya?tk=bM7haL For their future events see: https://ai-plans.com/about 

Further motivation: “Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well)”: https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for

A paper explaining the need and use for concave utility functions: “Using soft maximin for risk averse multi-objective decision-making”: https://link.springer.com/article/10.1007/s10458-022-09586-2 

Thanks for reading! If you have thoughts, questions, improvement suggestions, resource and collaborator references, feedback, or ideas, please share in the comments.