Recent discussions about artificial intelligence safety have focused heavily on ensuring AI systems remain under human control. While this goal seems laudable on its surface, we should carefully examine whether some proposed safety measures could paradoxically enable rather than prevent dangerous concentrations of power. The Control Paradox The fundamental tension...
This post is about risks in the development of increasingly capable AI, in particular the risk of losing control to AI and extinction risk. I'll suggest that a key question is, "When do we need to take this kind of risk seriously?" We'll look at the issue of 'agency overhang',...
how much earlier
Yeah, good question. I don't know really.
and does it matter?
I think so, because even if pure AI control follows on from human-AI entity control (which would actually be my prediction), I expect the dynamics of human-AI control to very much lead to and accelerate that eventual pure AI control.
I'm thinking, also, that there is a thing where pure AI entities need to be careful not to 'tip their hat'. What I mean by this is that pure AI entities will need to be careful not to reveal the extent of their capabilities up until a point where they are actually capable of taking control, whereas human-AI entities can kind of go ahead and play the power game and start to build up control without so much concern about this. (To the average voter, this could just look like more of the same.)
Is the sentence “in reality we should expect combined human-AI entities to reach dangerous capabilities before pure artificial intelligence” really true, and if so how much earlier and does it matter? (I lean towards “not necessarily true in the first place, and if true, probably not by much, and it’s not all that important”)
I guess in my model this is not something that suddenly becomes true at a certain level of capabilities. Instead, I think that the capabilities of human-AI entities become more dangerous in something of a continuous fashion as AI (and the technology for controlling AI) improves.
Recent discussions about artificial intelligence safety have focused heavily on ensuring AI systems remain under human control. While this goal seems laudable on its surface, we should carefully examine whether some proposed safety measures could paradoxically enable rather than prevent dangerous concentrations of power.
The fundamental tension lies in how we define "safety." Many current approaches to AI safety focus on making AI systems more controllable and aligned with human values. But this raises a critical question: controllable by whom, and aligned with whose values?
When we develop mechanisms to control AI systems, we are essentially creating tools that could be used by any sufficiently powerful entity - whether that's a government,... (read 1352 more words →)
In this blog post, I argue that a key feature we might be missing is that dangerous AI could potentially be a lot less capable than current state of the art LLMs, in some ways (specifically, less like a polymath): https://upcoder.com/21/is-there-a-power-play-overhang
(link post here: https://www.lesswrong.com/posts/7pdCh4MBFT6YXLL2a/is-there-a-power-play-overhang )
This post is about risks in the development of increasingly capable AI, in particular the risk of losing control to AI and extinction risk. I'll suggest that a key question is, "When do we need to take this kind of risk seriously?"
We'll look at the issue of 'agency overhang', which suggests that adding agent-like abilities to AI could result in a sudden and surprising increase in these kinds of risks.
I'll draw on intuitions about humans taking administrative and political control (with reference to the 'Dictator Book Club') and rephrase agency overhang as 'power play overhang'.
I'll finish by suggesting that a lot of people may be making a subtle but important mistake in imagining just one fairly specific path to dangerous AI.
There is the point about offensive/defensive asymmetry..