I think the problem of this analogy is assuming that if your competitors pull in the Trojan horse first, the Trojan horse will destroy them and then leave you alone.
I think this assumption is simply false in the AI race. If your competitors are racing, if you don't win the race, you just lose. If their AI turns out to be aligned, they win, and you lose (or if your goal is AI alignment and not power, you win anyway, but you still shouldn't give up the race in hopes of this happening unless you think your competitors are better at alignment than you are). If their AI is misaligned, then it screws over the world and you lose too.
I’ve argued that the AI situation is not clearly an ‘arms race’. By which I mean, going fast is not clearly good, even selfishly.
I think this is a hard point to get across. Like, these people are RACING. They say they are RACING. They are GOING FAST. If they stop RACING the other side will get there first. How is it not a RACE??
Which is a fair response.
It’s like if I said “this isn’t a chess tournament” gesturing at a group of chess champions aggressively playing chess. How could it not be?
Well, maybe all the prizes and recognition available in the circumstances are based on winning at checkers. That would make it, in a very important sense, not a chess tournament. They can play chess all they like, but it doesn’t make the incentive structure into that of a chess tournament. If they want to win at a tournament, their strategy is just badly mistaken.
It’s true that many people are trying to build AI very fast. But many people building AI very fast is different from being in a game where going very fast is the best selfish strategic move.
And this becomes important when “it’s really important to win at the race” becomes justification for a) moving fast at very high costs to other people, and b) giving up instead of trying to coordinate other players not to move fast, since other players are presumed to be immovably committed to winning the race due to that being so incentivized.
These justifications both require the structure of incentives to actually be a race, not just for people to be racing.
‘Is AI really an arms race or are people just racing?’ might sound like an abstract question. But if someone is saying they need to risk your family’s lives to fuel their quest to win an extremely high stakes chess championship, it’s very concretely important whether they are really in a chess championship!
While this is a basic point, my guess is that the distinction between what people are doing and what it is in their interests to do is too subtle and non-memorable to be tracked in the conversation.
So I propose an image I think might keep the incentives and the behavior separate more intuitively: AI as a Trojan horse race.
Various groups are working really hard to get various wooden horses through their own gates, resolute on doing so before their enemies pull in such a prize and outclass them with the contents. It’s an open question whether each horse contains fantastic treasure or a bunch of enemy agents. (This time in history we are even pretty confident that it includes a bunch of agents of some sort, and not at all confident of their loyalty..)
Is it enough to know that other cities are pulling horses through their gates? Are you satisfied then to have the biggest one pulled into your own town square?