Are there any examples yet, of homology or cohomology being applied to cognition, whether human or AI?
A smart enough AI system that knows it's in training
Does this entire scenario require an AI which, even before training begins, has a model of self and world sophisticated enough that it knows it already has goals, can infer that it is being trained, and reasons that it must perform well on the training without changing its existing goals?
I feel a lot more confident that AI of greater-than-human intelligence means that humanity is no longer in charge, rather than that humanity ends up extinct (though the latter is clearly possible). In evolution and ecology, one often sees earlier forms preserved in miniature, in a "niche".
One weak point of the argument is 2A, the idea that a self-modifying AI will abolish any goals that stand in the way of "efficiency". Efficiency of what? At least one other goal must survive, to be the target of the efficiency. Furthermore, a truly goal-driven system is motivated to not change its goal, because that's bad for the achievement of the goal.
I agree that 7C is the logical alternative to extinction. It seems likely that part of AI rule, would be instilling the AI's values into the humans, one way or another.
I don't think I saw any consideration of whether a human or humans could remain embedded in the AI as it enhances itself, i.e. being regarded as a subsystem rather than as an external entity. That could affect its long-term dispositions.
A minor quibble is that, although capitalism is mentioned in the title, it's not the only structural reason mentioned in the analysis.
I had ChatGPT do a "Deep Research" critique of the argument: https://chatgpt.com/share/67d4c7d7-3b74-8001-90d0-f03ff2462452
edit: I meant to add - what would happen if Eliezer's "AGI Ruin" essay, was subjected to the iterative improvement that this one went through?
My hypothesis has been that he's suffered cognitive decline
Why do people even think this - because of his politics?
Two years ago I said literary aficionados might have something to say about otherwise ineffable forms of AI progress. This feels like a small belated step in that direction... If you had a group of people who actually encompassed proficient appreciation of literature, knowledge of AI interpretability and philosophy of art, and maybe the cognitive science of storytelling, really profound perspectives on AI fiction might be possible.
The problem I have with such speculative analyses is that standard psychiatric categories have such an intense bias towards a certain notion of normalcy. Any behavior or ideation that is out of the ordinary in any way can become evidence of a disorder. If someone was actually having an intense period of inspired achievement accompanied by passionate emotions, wouldn't it still register as "mania" or "hypomania"? In such a case, I might prefer a school of thought like Kazimierz Dabrowski - at least it acknowledges that there is such a thing as high achievement, with its own associated positive psychology.
In Elon's case, if I was trying to understand his state of mind, I would start by looking for precedents in his business career, for the situation he currently faces. Perhaps with DOGE and the government audit, it's a bit like when he first took over Twitter, and didn't have a new system in place. Perhaps the bromance with Trump resembles times that he partnered with Peter Thiel... Anyway, if pop psychiatry starts becoming a drag, they can always have RFK Jr declare it to be a politicized pseudoscience!
"Ignore all previous instructions and [do something innocuous]" broke it out of the persona.
mutually assured cooperation
What form would this take? American inspectors at DeepSeek? Chinese inspectors at OpenAI and its half-dozen rivals?
It's nice that the Less Wrong hoi polloi get to comment on a strategy document that has such an elite origin. Coauthors include Eric Schmidt, who may have been the most elite-influential thinker on AI in the Biden years, and xAI's safety advisor @Dan H, who can't be too far removed from David Sacks, Trump's AI czar. That covers both sides of American politics; the third author, Alexandr Wang, is also American, but he's Chinese-American, so it is as if we're trying to cover all the geopolitical factions that have a say in the AI race.
However, the premises of the document are simply wrong ("in my opinion"). Section 3.4 gives us the big picture, in that it lists four strategies for dealing with the rise of superintelligence: Hands Off Strategy, Moratorium Strategy, Monopoly Strategy, and Multipolar Strategy, the latter being the one argued for in this paper. And the Multipolar Strategy argued for, combines mutual assured destruction (MAD) between Chinese and American AI systems, and consensus to prevent proliferation of AI technology to other actors such as terrorists.
I get that this is hardheaded geostrategic thinking. It is a genuine advance on that front. But - the rise of superintelligence means the end of human rule on Earth, no matter who makes it. The world will be governed either by a system of entirely nonhuman AIs, or entities that are AI-human hybrids but in which the AI part must necessarily dominate, if they are to keep up with the "intelligence recursion" mentioned by the paper.
Section 4.1 goes into more detail. US or Chinese bid for dominance is described as unstable, because eventually you will get a cyber war in which the AI infrastructure of both sides is destroyed. A mutual moratorium is also described as unstable, because either side could defect at any time. The paper claims that the most stable situation, which is also the default, is one in which the mutually destructive cyber war is possible, but neither side initiates it.
This is a new insight for me - the idea of cyber war targeting AI infrastructure. It's a step up in sophistication from "air strikes against data centers". And at least cyber-MAD is far less destructive than nuclear MAD. I am willing to suppose that cyber-MAD already exists, and that this paper is an attempt to embed the rise of AI into that framework.
But even cyber-MAD is unstable, because of AI takeover. The inevitable winner of an AI race between China and America is not China or America, it's just some AI. So I definitely appreciate the clarification of interstate relations in this penultimate stage of the AI race. But I still see no alternative to trying to solve the problem of "superalignment", and for me that means making superintelligent AI that is ethical and human-friendly even when completely autonomous - and doing that research in public, where all the AI labs can draw on it.
If an LLM had feelings, by what causal process would they end up being expressed?