We spent an evening at last week's Rationality Minicamp... We came up with a concrete (albeit greatly oversimplified) model...

Just to be clear: this model was drafted by a couple of mini-camp participants, not by the workshop as a whole, and isn't advocated by the Singularity Institute. For example, when I do my own back-of-the-envelopes I don't expect nearly a 30% increase in existential safety from convincing 30% of AI researchers that risk matters. Among other things, this is because there's a distance between "realize risk matters" and "successfully avoid creating UFAI" (much less "create FAI")", since sanity and know-how also play roles in AI design; and partly because there are more players than just AI researchers.

Still, it is good to get explicit models out there where they can be critiqued -- I just want to avoid folks having the impression that this is SingInst's model, or that it was taught at minicamp.