http://www.huffingtonpost.com/stephen-hawking/artificial-intelligence_b_5174265.html
Very surprised none has linked to this yet:
TL;DR: AI is a very underfunded existential risk.
Nothing new here, but it's the biggest endorsement the cause has gotten so far. I'm greatly pleased they got Stuart Russell, though not Peter Norvig, who seems to remain lukewarm to the cause. Also too bad this was Huffington vs something more respectable. With some thought I think we could've gotten the list to be more inclusive and found a better publication; still I think this is pretty huge.
There is no reason to expect exact equality, only close similarity. If you optimize, you still prefer something that's a tiny bit better to something that's a tiny bit worse. I'm not claiming that there is a significant difference. I'm claiming that there is some expected difference, all else equal, however tiny, which is all it takes to prefer one decision over another. In this case, a FAI gains you as much difference as available, minus the opportunity cost of FAI's development (if we set aside the difficulty in predicting success of a FAI development project).
(There are other illustrations I didn't give for how the difference may not be "tiny" in some senses of "tiny". For example, one possible effect is a few years of strongly optimized world, which might outweigh all of the moral value of the past human history. This is large compared to the value of millions of human lives, tiny compared to the value of uncontested future light cone.)
(I wouldn't give a Neanderthal as a relevant example of an optimizer, as the abstract argument about FAI's value is scrambled by the analogy beyond recognition. The Neanderthal in the example would have to be better than the fly at optimizing fly values (which may be impossible to usefully define for flies), and have enough optimization power to render the difference in bodies relatively morally irrelevant, compared to the consequences. Otherwise, the moral difference between their bodies is a confounder that renders the point of the difference in their optimization power, all else equal, moot, because all alse is now significantly not equal.)
Exactly. So for building FAI to be a good idea we need to expect its benefits to outweigh the opportunity cost (we can spend the remaining time "partying" rather than developing FAI).
Neat. One way it might work is the FAI running much-faster-than-realtime WBE's so that we gain a huge amount of subjective years of life. This works for any inevitable impending disaster.