Does anybody else on this board notice the similarities between speculations on the properties of AI and speculation on the properties of God? Will a friendly AI be able to protect us from unfriendly AIs if the friendly one is built first, locally?
Do we have strong evidence that we are NOT the paperclips of an AI? Would that be different from or the same as the creations of a god? Would we be able to tell the difference or would we only an observer outside the system be able to see the difference?
Does anybody else on this board notice the similarities between speculations on the properties of AI and speculation on the properties of God?
Why do you think Vernor Vinge dubbed AI, in one of his novels, "applied theology"?
Why do we imagine our actions could have consequences for more than a few million years into the future?
Unless what we believe about evolution is wrong, or UFAI is unlikely, or we are very very lucky, we should assume there are already a large number of unfriendly AIs in the universe, and probably in our galaxy; and that they will assimilate us within a few million years.
Therefore, justifications for harming people on Earth today in the name of protecting the entire universe over all time from UFAI in the future, like this one, should not be done. Our default assumption should be that the offspring of Earth will at best have a short happy life.
ADDED: If you observe, as many have, that Earth has not yet been assimilated, you can draw one of these conclusions:
Surely, for a Bayesian, the more reasonable conclusion is number 2! Conclusion 1 has priors we can estimate numerically. Conclusion 2 has priors we know very little about.
To say, "I am so confident in my beliefs about what a superintelligent AI will do, that I consider it more likely that I live on an astronomically lucky planet, than that those beliefs are wrong", is something I might come up with if asked to draw a caricature of irrationality.