Heading off a near-term AGI arms race

lincolnquirk

I know people have talked about this in the past, but now seems like an important time for some practical brainstorming here. Hypothetical: the recent $15mm Series A funding of Vicarious by Good Ventures and Founders Fund sets off a wave of $450mm in funded AGI projects of approximately the same scope, over the next ten years. Let's estimate a third of that goes to paying for man-years of actual, low-level, basic AGI capabilities research. That's about 1500 man-years. Anything which can show something resembling progress can easily secure another few hundred man-years to continue making progress.

Now, if this scenario comes to pass, it seems like one of the worst-case scenarios -- if AGI is possible today, that's a lot of highly incentivized, funded research to make it happen, without strong safety incentives. It seems to depend on VCs realizing the high potential impact of an AGI project, and of the companies having access to good researchers.

The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:

...I think this exactly the sort of innovation timeline real venture capitalists should be considering - funding real R&D that could have a revolutionary impact even if the odds are against it.

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?

I'll admit to being very scared.

The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:

...I think this exactly the sort of innovation timeline real venture capitalists should be considering - funding real R&D that could have a revolutionary impact even if the odds are against it.

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?

I'll admit to being very scared.

Because it's bad tactics to endorse it in the open or because sabotaging unfriendly AI research is a case of not even if it's the right thing to do?

I assume you'd slow down or kibosh a not-proved-to-be-friendly AGI project if you had the authority to do so. But you wouldn't interfere if you didn't have legitimate authority over the project? There are plenty of not-illegal, but still ethical norm-breaking opportunities for sabotage, (deny the people on the project tenure, if you're in a university, hire the best researchers away, etc).

Do you think this shouldn't be discussed out of respect for the law, out of respect for the autonomy of researchers, or a mix of both?

Intelligence does not imply benevolence. Surely, there already are people who will try to sabotage unFriendly projects.

8Xachariah14y

If we were certain a uFAI were going online in a matter of days, it would be everyone's responsibility to stop it by any means possible. Imminent threat to humanity and all that. However, it's a very low probability that it'll ever get to that point. Talking about and endorsing (hypothetical) unethical activity will impose social costs in the meanwhile. So, it's a net negative to discuss it.

6jmmcd14y

It might damage LW's credibility among decision-makers and public opinion. Of course, it might improve LW's credibility among certain other groupings. PR is a tricky balancing act. PETA is a good example in both cases.

10

Heading off a near-term AGI arms race

10

10

10

Heading off a near-term AGI arms race

10

10