Heading off a near-term AGI arms race

lincolnquirk

I know people have talked about this in the past, but now seems like an important time for some practical brainstorming here. Hypothetical: the recent $15mm Series A funding of Vicarious by Good Ventures and Founders Fund sets off a wave of $450mm in funded AGI projects of approximately the same scope, over the next ten years. Let's estimate a third of that goes to paying for man-years of actual, low-level, basic AGI capabilities research. That's about 1500 man-years. Anything which can show something resembling progress can easily secure another few hundred man-years to continue making progress.

Now, if this scenario comes to pass, it seems like one of the worst-case scenarios -- if AGI is possible today, that's a lot of highly incentivized, funded research to make it happen, without strong safety incentives. It seems to depend on VCs realizing the high potential impact of an AGI project, and of the companies having access to good researchers.

The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:

...I think this exactly the sort of innovation timeline real venture capitalists should be considering - funding real R&D that could have a revolutionary impact even if the odds are against it.

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?

I'll admit to being very scared.

The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:

...I think this exactly the sort of innovation timeline real venture capitalists should be considering - funding real R&D that could have a revolutionary impact even if the odds are against it.

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?

I'll admit to being very scared.

However, it's a very low probability that it'll ever get to that point.

What specifically do you consider low probability? That an uFAI will ever be launched, or that there will be an advance high credibility warning?

Sorry, I was imprecise. I consider it likely that eventually we'll be able to make uFAI, but unlikely that any particular project will make uFAI. Moreover, we probably won't get appreciable warning for uFAI because if researchers knew they were making a uFAI then they wouldn't make one.

Thus, we have to adopt a general strategy that can't target any specific research group. Sabotage does not scale well, and would only drive research underground while imposing social costs on us meanwhile. The best bet then is to promote awareness of uFAI risks and try t... (read more)

13gwern14y

I'd argue the latter. It's hard to imagine how you could know in advance that a uFAI has a high chance of working, rather than being one of thousands of ambitious AGI projects that simply fail. (Douglas Lenat comes to you, saying that he's finished a powerful fully general self-modifying AI program called Eurisko, which has done very impressive things in its early trials, so he's about to run it on some real-world problems on a supercomputer with Internet access; and by the way, he'll be alone all tomorrow fiddling with it, would you like to come over...)

10

Heading off a near-term AGI arms race

10

10

10

Heading off a near-term AGI arms race

10

10