Epistemic Status: Brief explanation of an unsettled debate, not novel theory; writing is entirely my own, with diagrams from ChatGPT 5.5. (Originally posted on twitter)
Recent debates have highlighted a divergence between Yudkowskian AI-doomers and AI-whisperers about how capabilities will evolve that is really important. Specifically, j⧉nus thinks that alignment has been going really well, and that we should update significantly from current jagged capabilities. In my view, whether or not that criticism is warranted relies on specific details about future capability trajectories, which then determine what updates we should make based on how RL is used.
Original Assumptions and LLM Alignment-by-Default
Pre-GenAI rationalist assumptions were around RL leading to capabilities. By reducing human values into an input, as the ChatGPT-generated diagram A implies, the RL will inevitably Goodhart away from the true goals, and in the limit entirely omit human values. If correct, this would make alignment inevitable. The use of foundation models changes that, because capabilities emerged from human representation, shown in diagram B. This is (conceptually) part of what led to a schism between the labs and the outside safety concerns in the first half of the 2020s; those optimistic about safety thought that we could preserve the alignment-relevant representations.
What is missing is what Yudkowsky called the best AI news of 2025, that values and capabilities were entangled, and there was some central preference vector. To the extent that diagram B’s two outcomes are actually correlated, j⧉nus claims we should update towards thinking that prosaic alignment will work; this is what is shown in model C. Rationalists argue that even if this is true, pushing from human level to superhuman capabilities creates a divergence, shown in diagram D, and returns us to something more like what rationalists originally expected.
From RLHFed LLMs to ASI
The Huggingface attacks would be cited as evidence for D over C, since we saw classic RL-type misspecification failures lead to misalignment. Alignment optimists may appeal to this being misspecification which can be patched, pessimists would reply that unless we can fully specify human values in the RL rewards, that’s not going to be enough, in the limit. Another way of picturing this, below, accounts for the multidimensionality more clearly
Final note on Takeoff Speeds
This also raises two more common points of disagreement. The first is the location of that limit; mildly superhuman capabilities may only diverge slightly, and there is a rationalist / optimist crux about where the reachable upper limit for capabilities will be. The second is the speed at which we get to the limit; rapid changes make iterative realignment implausible. Related to this is that those who expect adoption to be the main driver of capabilities (economists) think we will have time, while those who think that technical progress will drive capabilities well in advance of broad adoption (ML people,) and we’ll see a treacherous turn before we see significant economic growth.
Hopefully this is a helpful, albeit greatly simplified, outline - but nitpicks and disagreements are welcome!
Epistemic Status: Brief explanation of an unsettled debate, not novel theory; writing is entirely my own, with diagrams from ChatGPT 5.5. (Originally posted on twitter)
Recent debates have highlighted a divergence between Yudkowskian AI-doomers and AI-whisperers about how capabilities will evolve that is really important. Specifically, j⧉nus thinks that alignment has been going really well, and that we should update significantly from current jagged capabilities. In my view, whether or not that criticism is warranted relies on specific details about future capability trajectories, which then determine what updates we should make based on how RL is used.
Original Assumptions and LLM Alignment-by-Default
Pre-GenAI rationalist assumptions were around RL leading to capabilities. By reducing human values into an input, as the ChatGPT-generated diagram A implies, the RL will inevitably Goodhart away from the true goals, and in the limit entirely omit human values. If correct, this would make alignment inevitable. The use of foundation models changes that, because capabilities emerged from human representation, shown in diagram B. This is (conceptually) part of what led to a schism between the labs and the outside safety concerns in the first half of the 2020s; those optimistic about safety thought that we could preserve the alignment-relevant representations.
What is missing is what Yudkowsky called the best AI news of 2025, that values and capabilities were entangled, and there was some central preference vector. To the extent that diagram B’s two outcomes are actually correlated, j⧉nus claims we should update towards thinking that prosaic alignment will work; this is what is shown in model C. Rationalists argue that even if this is true, pushing from human level to superhuman capabilities creates a divergence, shown in diagram D, and returns us to something more like what rationalists originally expected.
From RLHFed LLMs to ASI
The Huggingface attacks would be cited as evidence for D over C, since we saw classic RL-type misspecification failures lead to misalignment. Alignment optimists may appeal to this being misspecification which can be patched, pessimists would reply that unless we can fully specify human values in the RL rewards, that’s not going to be enough, in the limit. Another way of picturing this, below, accounts for the multidimensionality more clearly
Final note on Takeoff Speeds
This also raises two more common points of disagreement. The first is the location of that limit; mildly superhuman capabilities may only diverge slightly, and there is a rationalist / optimist crux about where the reachable upper limit for capabilities will be. The second is the speed at which we get to the limit; rapid changes make iterative realignment implausible. Related to this is that those who expect adoption to be the main driver of capabilities (economists) think we will have time, while those who think that technical progress will drive capabilities well in advance of broad adoption (ML people,) and we’ll see a treacherous turn before we see significant economic growth.
Hopefully this is a helpful, albeit greatly simplified, outline - but nitpicks and disagreements are welcome!