I don't think we can push much beyond that before a system will figure out pretty much everything important about itself.

Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an "alignment tax" there can be a "lack of self awareness or meta-cognition penalty". While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.

Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.

Replying toHuman-like metacognitive skills will reduce LLM slop and aid alignment and capabilities

RussellThor1d

Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities

metacognitive skills

Why not directly call it self awareness? I notice you mention self awareness later as part of it - is there a reason for not using a common term.
I say this because there doesn't seem to be investigation specifically into this topic, instead it seems to be actively avoided for some reason. "Situational awareness" is often used instead etc.

Additionally I expect there is a basin of attraction around what we intuitively mean by the term "self awareness". A self has obvious evolutionary benefit, and I expect that transfers to AI. So that means just like instrumental convergence, models could converge to self awareness as we commonly understand it. We often use... (read more)

Replying toIABIED Book Review: Core Arguments and Counterarguments

RussellThor9d

IABIED Book Review: Core Arguments and Counterarguments

An objection not mentioned here that I saw from Emmett Shear was in general against assuming the AI will misbehave out of distribution. That isn't explicitly mentioned here, but is often in the literature, especially older texts such as Superintelligence by Bostrom. His argument was that generalizing OOD well is an essential feature of intelligence and any superintelligence will in fact do it better than humans.

Replying toThe false confidence theorem and Bayesian reasoning

RussellThor1mo

The false confidence theorem and Bayesian reasoning

This raises a good point, especially with ethics and blame/credit. Before the Rootclaim debate, I gave a lab leak at about 65% confidence. After the debate and some thought, I put it much lower, <10% with a lot of my uncertainty based on my relatively low biology knowledge. If someone asks, I may say ~about 10% but I'm not enough of an expert. (I also expect if I was more of an expert or spent a lot more time, my 10% would go down, in tension to rationality)

HOWEVER that does not mean I can ethically judge the GoF researchers as if they had taken a 10% chance at killing >20 million people... (read more)

RussellThor1mo

I have raised this twice with UKR over the last year. Surprised they havn't done it yet.

Replying toTwo Aspects of Situational Awareness: World Modelling & Indexical Information

RussellThor1mo

Two Aspects of Situational Awareness: World Modelling & Indexical Information

Suppose you know everything about the past, present, and future of the universe in complete physical detail, but you don't know where YOU are in space-time

You can't know the future in your example because the agent "You" can change it. You can only know the past and the light cone that cannot be affected by your actions. That then precisely locates you in space and time. For example if your first action is to use a fully random number generator, then knowledge of the future affected by that is impossible. (You could also scan the entire past/present/future for a string/pattern that does not happen and broadcast it, also breaking the assumption)

Beliefs and position going into 2026

RussellThor

1mo

At least every year now I will state my current position, reflect on that I had a year ago, check predictions etc.
Here is last years article and 6 month update. I haven't changed my position on anything major.

Main points

Transformer architecture won't get to TAI (Transformative AI), however it may get close.

This is probably my most important prediction and it is based mostly on data efficiency. Quite simply, Einstein did not need to read the entire human literature to be creative. Current AI needs far more data than humans. LLM and self driving both follow this pattern, I believe it is a fundamental limitation of the current architecture. LLM may however speed up... (read 1414 more words →)

Replying to2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

RussellThor2mo

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

How much do you think this will matter for AGI/ASI? e.g. here

Will the RL training method have to be very different when self awareness comes into play? Are our current RL techniques sample efficient enough to get to ASI?

Replying toThe behavioral selection model for predicting AI motivations

RussellThor2mo

The behavioral selection model for predicting AI motivations

Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the optimization target, and instead reward “chisels” a combination of context-dependent cognitive patterns into the AI.

Increased self awareness could change this.

You can think of a scale, where rewards chiseling cognitive patterns is at one end. That is reward happens to the AI without it being aware that such a thing even exists. Think Alpha-go type AI. Then there is the AI knowing enough about reward to potentially pursue it, but not thinking further about what this means.

As others have said, not covered by this article are things like... (read more)

GPT fail leads to shorter timelines?

If you are of the opinion that the transformer architecture cannot scale to AGI and a more brain inspired approach is needed, then the sooner that everyone realizes that scaling LLM/Tx is not sufficient, the sooner the search begins in earnest. At present the majority of experimental compute and researcher effort is probably on such LLM/Tx systems, however if that changes to exploring new approaches then we can expect a speedup on finding such better architectures.
For existing companies, https://thinkingmachines.ai/ and https://ssi.inc/ are probably already doing a lot of this, and Deepmind is not just transformers, but there is a lot of scope for effort/compute to shift from LLM to other ideas in the wider industry.

Anonymous bulk age verification

As age verification comes common with all its privacy and govt overreach downsides is there a good way to balance privacy and proof of age?

I support an R16-18 ban for the worst social media platforms (no NOT Spotify/Wikipedia!!) but am quite concerned about recent privacy/"safety" behaviors by governments lately.

I consider a system somewhat similar to how voting works in our country. You verify yourself, then choose from random a voting slip, breaking the link from ID to how you vote.
Let say Facebook wanted to do this worldwide. They create millions of sealed ID verified tokens. This is an envelope with a tamper proof seal and instructions to only open... (read more)

The anti-Kardashev scale is a better measure of civilizational power

RussellThor

7mo

This post is making a point that would appear to be obvious, however given how the Kardashev scale and direct energy usage comes up often in popular discussion, it seems it needs to be made. The Kardashev scale is a poor measure of civilizations progress and power.

The Kardashev scale is not at all even in terms of difficulty

An essential feature a scale must have to be useful is that the levels on it must be somewhat evenly spaced. The Kardashev scale does not have that, and I will stretch the point here by arguing that K3, harnessing the power of a galaxy is in fact below K2 (star) from one point of... (read 637 more words →)

Beliefs and state of mind into 2025

RussellThor

This post is to record the state of my thinking at the start of 2025. I plan to update these reflections in 6-12 months depending on how much changes in the field of AI.

1. Summary

It is best not to pause AI progress until at least one major AI lab achieves a system capable of providing approximately a 10x productivity boost for AI research, including performing almost all tasks of an AI researcher. Extending the time we remain in such a state is critical for ensuring positive outcomes.

If it was possible to stop AI progress sometime before that and focus just on mind uploading, that would be preferable, however I don’t think that... (read 1823 more words →)

Dishbrain and implications.

RussellThor

I believe that AI research has not given sufficient attention to learning directly from biology, particularly through the direct observation and manipulation of neurons in controlled environments. Furthermore, even after learning all that biology has to offer, neurons could still play a part in the post TAI world economy as they could be cheaper and faster to grow than chips are to manufacture.

Pre TAI – study neurons to greatly increase learning capability

As I have said in other places on this site, I believe that the current transformer architecture will not scale to TAI, because it does not learn fast enough or generalize well enough from data compared to biology. For example, Tesla... (read 467 more words →)

Vision of a positive Singularity

RussellThor

Introduction

Many people feel significant anxiety about how AI and superintelligence will play out, regarding both the ultimate outcome and the intermediate stages. There is the sense that some kind of loss is inevitable as humanity becomes more powerful in this way. There is also the concern that there will be no place for a person or society with existing values any more as things progress. I try to think up a plan and system of values that will respect everyone's desires as much as possible. The goal is to ensure coexistence among groups with differing values, minimizing conflict over competing visions for the future.

A clear positive vision is important in times of... (read 1171 more words →)

Types of takeoff

When I first heard and thought about AI takeoff I found the argument convincing that as soon as an AI passed IQ 100, takeoff would become hyper exponentially fast. Progress would speed up, which would then compound on itself etc. However there other possibilities.

AGI is a barrier that requires >200 IQ to pass unless we copy biology?

Progress could be discontinuous, there could be IQ thresholds required to unlock better methods or architectures. Say we fixed our current compute capability, and with fixed human intelligence we may not be able to figure out the formula for AGI, in a similar way that the combined human intelligence hasn't cracked many hard problems... (read 618 more words →)

Putting down a prediction I have had for quite some time.
The current LLM/Transformer architecture will stagnate before AGI/TAI (That is the ability to do any cognitive task as effectively and cheaper than a human)

From what I have seen, Tesla autopilot learns >10,000 slower than a human datawise.

We will get AGI by copying nature, at the scale of a simple mammal brain, then scaling up, like this kind of project:

https://x.com/Andrew_C_Payne/status/1863957226010144791
https://e11.bio/news/roadmap
I expect AGI to be 0-2 years after a mammal brain is mapped. In terms of cost-effectiveness I consider such a connectome project to be far more cost effective per $ than large training runs or building a 1GW data center etc if you goal is to achieve AGI.

That is TAI by about 2032 assuming 5 years to scan a mammal brain. In this case there could be a few years when Moores law has effectively stopped, larger data centers are not being built and it is not clear where progress will come from.

Of Birds and Bees

RussellThor

The Hierarchy

There is a hierarchy in life from simple cells to complex cells to multi-cellular creatures to creatures that often live in a groups like birds and bees. As we go up the hierarchy the lower levels have less individual say. For example the mitochondria have no individual life or fitness outside of the cell, and the cells success is not meaningfully measured outside that of the organism in multi-cellular creatures.

Generally as we go up the hierarchy the higher level is more complex and smarter than the lower level. A brain can think, a neuron cannot. A hive of bees or ants is more capable than an individual.

Bird patterns

However you see sometimes... (read 403 more words →)

Brilliant Pebbles?

See here and here

This idea has come back up, and it could be feasible this time around because of the high launch capability and total reusability of SpaceX's Starship. The idea is a large constellation (~30,000?) of low earth satellites that intercept nuclear launches in their boost phase where they are much slower and more vulnerable to interception. The challenge of course is that you constantly need enough satellites overhead at all times to intercept the entire arsenal of a major power if they launch all at once.

There are obvious positives and risks with this

The main positive is it removes the chance of a catastrophic nuclear war.

Negatives are potentially destabilizing the... (read 612 more words →)

How does intelligence scale with processing power

A default position is that exponentially more processing power is needed for a constant increase in intelligence.
To start, lets assume a guided/intuition + search model for intelligence. That is like Chess or Go where you have an evaluation module and a search module. In simple situations an exponential increase in processing power usually gives a linear increase in lookahead ability and rating/ELO in games measured that way.

However does this match reality?

What if the longer the time horizon, the bigger the board became, or the more complexity was introduced. For board games there is usually a constant number of possibilities to search at every ply of lookahead... (read 449 more words →)

Is X.AI currently performing the largest training run?
This source claims it is

PC mag not so sure here, here

If so it seems to be getting a lot less attention compared to its compute capability.

Not sure if I have stated this before clearly but I believe scaling laws will not hold for LLM/Transformer type tech, and at least one major architectural advance is missing before AGI. That is increasing scaling of compute and data will plateau performance soon, and before AGI. Therefore I expect to see evidence for this not much after the end of this year, when large training runs yield models that are a lot more expensive to train, slower on inference and only a little better on performance. X.AI could be one of the first to publicly let this be known (Open AI, etc could very well be aware of this but not making it public)

Evaluation vs Symbolism

TLDR
Thinking about the Busy Beaver numbers has lead me to believe that just because a theorem holds true for a massive number of evaluated examples, this is only weak evidence it is actually true. Can we go meta on this?

Main
After reading a post by Scott Aaronson, and this coming to my attention https://en.wikipedia.org/wiki/Prime_number_theorem and Littlewood's theorem

"Li(𝑥) overestimates the number of primes below x more often than not, especially as x grows large. However, there are known low-lying values (like around x=10^316) discovered by Littlewood) where 𝜋(𝑥) exceeds Li(x), contradicting the general trend."

This got me thinking about how common this kind of thing is and why? Why does a formula hold... (read 518 more words →)

Technology is about making boring stuff non-conscious. Beginning from basic physical movement such as making a wheel go round, to arithmetic and now code snippets that are so commonly used they shouldn't require re-thinking. This is a reason why AI art upsets people - we actually want that to be the result of a conscious process. If you make boring stuff that creates power or wealth non-conscious then everyone is happier. Meat production would be much better if it was non-conscious. The more AI is non-conscious for a given level of capability, the better off we are.

Self modelling in NN https://arxiv.org/pdf/2407.10188 Is this good news for mech interpretability? If the model makes it easily predictable, then that really seems to limit the possibilities for deceptive alignment

Replying toA "Bitter Lesson" Approach to Aligning AGI and ASI

RussellThor2y

A "Bitter Lesson" Approach to Aligning AGI and ASI

Yes agreed - is it possible to make a toy model to test the "basin of attraction" hypothesis? I agree that is important.

One of several things I disagree with the MIRI consensus is the idea that human values are some special single point lost in a multi-dimensional wilderness. Intuitively the basin of attraction seems much more likely as a prior, yet sure isn't treated as such. I also don't see data to point against this prior, what I have seen looks to support it.

Further thoughts - One thing that concerns me about such alignment techniques is that I am too much of a moral realist to think that is all you need.... (read more)

Replying toDrone Wars Endgame

RussellThor2y

Drone Wars Endgame

OK firstly if we are talking fundamental physical limits how would sniper drones not be viable? Are you saying a flying platform could never compensate for recoil even if precisely calibrated before? What about fundamentals for guided bullets - a bullet with over 50% chance of hitting a target is worth paying for.

Your points - 1. The idea is a larger shell (not regular sized bullet) just obscures the sensor for a fraction of a second in a coordinated attack with the larger Javelin type missile. Such shell/s may be considerably larger than a regular bullet, but much cheaper than a missile. Missile or sniper size drones could be fitted with such... (read more)

Drone Wars Endgame

RussellThor

Note: Updates at end of article on 9 Dec 2025 to reflect recent changes

1. Introduction

It is probably becoming clear to anyone who follows war lately, especially the Ukraine conflict that automatic weapons, especially drones are rapidly changing what weapons and tactics are effective.

The purpose of this article is to first consider how a fully autonomous force with current or near term tech would go against a state of the art conventional force, then to consider where the equilibrium would be on attack vs defense for autonomous forces using advanced but foreseeable tech.

1.1 Description of fully autonomous force using near term tech

The idea is to have a force with as few different types... (read 2700 more words →)

Little attention seems to be on discouraging hardware progress

RussellThor

There is a lot of attention being given to AI software progress, potential regulation etc., however articles such as this celebrating potential progress towards a step change in AI ability seem to get little attention and a free pass in the press as being uncontroversial.

As someone who believes a slow takeoff is likely (though not with enough certainty to be comfortable), I would like to see more emphasis placed on the potential disruption from hardware advancements. I contend that with our current production capacity of roughly 50 million GPUs per year, and given their current specifications, it is unlikely that they could pose an existential risk, regardless of the level of software... (read more)

P-zombies, Compression and the Simulation Hypothesis

RussellThor

Can we have approximate p-zombies?

Lets say we accept that p-zombies don't make sense because of the core arguments. However what about a creature who's behavior is very similar to the target but with a different neural software? That is a simulated person who's behavior is different only by a small, perhaps undetectable amount from the real person, but with internal software so different that we would not expect them to have the same consciousness, if any at all?

Approximate p-zombies would be very useful for Simulations

Say you want to run an ancestor simulation. Your goal is to have the resulting world turn out as accurate as possible after a number of years. This... (read 1341 more words →)

LESSWRONG
LW

LESSWRONG
LW

RussellThor

Drone Wars Endgame

Beliefs and state of mind into 2025

Has the Symbol Grounding Problem just gone away?

Of Birds and Bees

RussellThor

RussellThor

Beliefs and position going into 2026

The anti-Kardashev scale is a better measure of civilizational power

Beliefs and state of mind into 2025

Dishbrain and implications.

Vision of a positive Singularity

RussellThor's Shortform

Of Birds and Bees

RussellThor

Drone Wars Endgame

Beliefs and state of mind into 2025

Has the Symbol Grounding Problem just gone away?

Of Birds and Bees

RussellThor

RussellThor

Beliefs and position going into 2026

The anti-Kardashev scale is a better measure of civilizational power

Beliefs and state of mind into 2025

Dishbrain and implications.

Vision of a positive Singularity

RussellThor's Shortform

Of Birds and Bees

Main points

The Kardashev scale is not at all even in terms of difficulty

1. Summary

Note: Updates at end of article on 9 Dec 2025 to reflect recent changes

1. Introduction