I mentioned that people inside such projects tend to claim that the projects are more safe than outsiders, which is not surprising. The same is true about AI research - most concerns about AI safety come from outsiders, not actual researchers.
I think that insiders tend to underestimate risks because they interested in continuation of the project and because of selection effect - those who do not think that it is wise to proceed with a project already leaved it. It is like driving - everybody could see that someone is reckless driver, but he is sure that he is ok.
Outsiders tend to overestimate risks, as they may not see inside safety mechanisms.
In general I think that we need independent safety committee to evaluate safety of all projects. And I oppose any project including LHC, which safety was not estimated BEFORE its start by independent body.
Keep in mind that independent safety committee's are not free. Suddenly everything costs more time, money and energy.
Does my friends AI-based masters thesis on translation need to be evaluated?
Realistically we're at more risk of the worlds ants suddenly turning on us than his translation code but at this point you have an Independent Safety Committee and the precept that AI related projects need to be assessed.
Suddenly he'd find that he'd have to spend weeks preparing applications and reports for the committee and if they're inept they could kill his wh...
Articles covering the ideas of inside view and outside view:
Beware the Inside View (by Robin Hanson)
Outside View LessWrong wiki article
Article discussing the weighting of inside view and outside view:
The World is Mad (by ozymandias)
A couple of potential extinction events which seem to be most easily mitigated (the machinery involved is expensive):
Broadcasting powerful messages to the stars:
Should Earth Shut the Hell Up? (by Robin Hanson)
Arecibo message (Wikipedia)
Large Hadron Collider:
Anyone who thinks the Large Hadron Collider will destroy the world is a t**t. (by Rebecca Roache)
How should the inside view versus the outside view be weighted when considering extinction events?
Should the broadcast of future Arecibo messages (or powerful signals in general) be opposed?
Should the expansion of energy levels (or continued operation at all) of the Large Hadron Collider be opposed?