Logos, you don't need to preach about utilitarian calculations to us. You have it the other way around. We don't condemn your words because we can't make them, we condemn them because we can make them better than you.
Then do so.
That you don't seem to consider that, nor do you urge others to consider it, is part of the fatal irresponsibility of your words.
I don't seem to consider it because it is a necessary part of the calculus of determining whether a belief is valid. This would be why I mentioned "material evidence" at all -- an indicator that checks and confirmations are necessary to a sufficiently rigorous epistemology. The objection of "but it could be a faulty belief" is irrelevant here. We have already done away with it in the formation of the specific counterfactual. That it is an exceedingly unlikely counterfactual does not change the fact that it is a useful counterfactual.
What I'm elucidating here is a rather ugly version of a topic that Eliezer was discussing with his Sword of Good parable: to be effective in discerning what is morally correct one must be in the practice and habit of throwing away cached moral beliefs and evaluating even the most unpleasant of situations according to their accepted epistemological framework's methodology for such evaluations.
The AI serial-killer scenario is one such example.
I don't seem to consider it because it is a necessary part of the calculus of determining whether a belief is valid. This would be why I mentioned "material evidence" at all -- an indicator that checks and confirmations are necessary to a sufficiently rigorous epistemology.
Don't you think that a remotely responsible post should have at the very least emphasized that significantly more than you did?
If tomorrow some lone nut murders an AI researcher, and after being arrested says they found encouragement in your specific words, and also says th...
Here's a poser that occurred to us over the summer, and one that we couldn't really come up with any satisfactory solution to. The people who work at the Singularity Institute have a high estimate of the probability that an Unfriendly AI will destroy the world. People who work for http://nuclearrisk.org/ have a very high estimate of the probability that a nuclear war will destroy the world (by their estimates, if you are American and under 40, then nuclear war is the single most likely way in which you might die next year).
It seems like there are good reasons to take these numbers seriously, because Eliezer is probably the world expert on AI risk, and Hellman is probably the world expert on nuclear risk. However, there's a problem - Eliezer is an expert on AI risk because he believes that AI risk is a bigger risk than nuclear war. Similarly, Hellman chose to study nuclear risks and not AI risk I because he had a higher than average estimate of the threat of nuclear war.
It seems like it might be a good idea to know what the probability of each of these risks is. Is there a sensible way for these people to correct for the fact that the people studying these risks are those that have high estimate of them in the first place?