If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "
Okay. A good metaphor might be the development of the atomic bomb. Lots of nuclear physicists thought that nuclear reactions couldn't be used for useful energy (e.g. Rutherford). Leo Szilard had the insight that you could do a chain reaction and that this might be dangerous. He did not build it the bomb (he could not, he didn't know about neutrons) and assigned the patent to the admiralty to keep it secret.
But he managed to convince other high profile physicists that it might dangerous without publicizing it too much (no whitepapers etc). He had the reputation etc and the physics of these things was far more firm than our whispy grasp of intelligence.
So that worked.
But how will it work for our hypothetical AI researcher who has the breakthrough, if they are not part of the in group of ai risk people? They might be chinese and not have a good grasp on english. They are motivated to try and get the word to say Elon Musk (or another influential concerned person/group that might be able to develop it safely) of their breakthrough but want to keep the idea as secret as possible and do not have the pathway of reaching them.
One issue is that you're judging the idea of a chain reaction as a breakthrough post factum. At the time, it was just a hypothesis, interesting but unproven. I don't know the history of nuclear physics well enough, but I suspect there were other hypotheses, also quite interesting, which didn't pan out and we forgot about them.
A breakthrough idea is by definition weird and doesn't fit into the current paradigm. At the time it's proposed, it is difficult to separate real breakthroughs from unworkable craziness unless you can demonstrate that your breakthroug... (read more)