Help good guys beat the race:
Please provide constructive criticism.
An open source project might prevent this problem, not because having an open source AGI is safe, but because 1.) open source projects are open, so anybody can influence it, including people who are knowledgeable about risks and 2.) the people involved in open source projects probably tend to have a pretty strong philanthropic streak and they're more likely to listen to the dangers than a risk-taking capitalist. The reason it may stop them is this: If an open source project gets there first, it won't be seen as a juicy target to capitalists anymore. It will be a niche that's already filled for free. If they wanted to make an AGI they'd have to make one that was so much better than the existing one that it makes sense to charge, or fail at business.
Making an open source AGI, in order to compete with a business might cause the open source programmers to rush. However, imagine what would happen if customers got the following messages around the time that the closed source AGI was going to be released: If you wait a while longer, an AGI will come out for free, plus, the open source AGI is going to be thoroughly tested to discover dangers before you run it. The closed source AGI is very risky." That would deter a lot of people from buying, which would at least reduce the exposure to the closed source AGI - and the open source group would not have to release the AGI until they had tested thoroughly. If, during the course of their tests, they discover hideous risks, these could serve as warnings about AGI in general, make those risks feel real, and prevent people from running risky AGIs. Assuming that the open source project had good PR and advertising / public education campaigns.
Why open source might have a competitive advantage:
Open source people may be more willing to merge, especially if our future depends on it, whereas companies tend to behave in self-interested ways and work separately for the most part. They're already divided, so open source could conquer them.
I was told by a Microsoft employee that he thought Linux would eventually win. Considering the influence that corporate culture can have on software design (the rushing to make deadlines which results in code debt), I don't disagree with him one bit. One concept here that could turn out to be really important is that any company working on AGI that does not put safety first may also have a short-term culture, which means they might actually take much, much longer to release their project, or to have recalls that force them to start over, than an organization of programmers that is allowed to do things the right way. An open source project has that potential benefit on it's side.
People who work on open source projects are probably more altruistic. They may be able to be persuaded that working on AI is so much more important to the future of humanity that they jump out of their current open source project and get involved.
For those three reasons, I think an open source project has a good chance of getting there first.
The obvious argument against this would be "An open source AGI!!! Won't bad people write their own versions?" My counter argument is: In a world where pirates routinely crack software within days of it coming out, and corporate espionage is a real possibility for a target this juicy, what makes you think the code won't get stolen THE VERY NEXT DAY? In that event, the best tool to save us from rogue AGIs would be if every open source programmer has access to editable copies of a friendly AGI, don't you think?
An even faster solution: How just the threat of having to compete with a massive open source project may stop them.
See Also "Sabotage would not work"
Analogizing AGI mainly to existing software projects probably isn't a good starting point for an useful contribution. The big problems are mostly tied to the unique features an actual AGI would have, not to making a generic software project with some security implications work out right.
For a different analogy, think about a software that fits on a floppy disk that somehow turns any laptop into an explosive device with a nuclear bomb level yield (maybe it turns out you can set up a very specific oscillation pattern in a multicore CPU silicon that will trig...
I know people have talked about this in the past, but now seems like an important time for some practical brainstorming here. Hypothetical: the recent $15mm Series A funding of Vicarious by Good Ventures and Founders Fund sets off a wave of $450mm in funded AGI projects of approximately the same scope, over the next ten years. Let's estimate a third of that goes to paying for man-years of actual, low-level, basic AGI capabilities research. That's about 1500 man-years. Anything which can show something resembling progress can easily secure another few hundred man-years to continue making progress.
Now, if this scenario comes to pass, it seems like one of the worst-case scenarios -- if AGI is possible today, that's a lot of highly incentivized, funded research to make it happen, without strong safety incentives. It seems to depend on VCs realizing the high potential impact of an AGI project, and of the companies having access to good researchers.
The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:
Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?
I'll admit to being very scared.