I'd like to hear some justification - some extensive justification, at least a sequence's worth - explaining how building a Friendly AI, with the already-expressed intent of beating all other AIs to the punch and then using your position of power to suppress or destroy construction of any other AIs at any cost, and to make yours a singleton designed in such a way that the values you programmed it with can never be altered -
-- can amount to anything other than what Robin just described.
(Elaborating after a day with no responses)
I realize that the first answer is going to be something along the lines of, "But we don't program in the values. We just design an algorithm that can extrapolate values from everyone else."
First, I've spoken with many of the people involved, and haven't heard any of them express views consistent with this - they want their values to be preserved, in fact two said explicitly that they did not care what happened to the universe if their personal values were not preserved - and yet they also believe that their values are extreme minority views among humanity. What's more, they have views of transhumanism and humanity that make excluding lower animals from this extrapolated volition unjustifiable on any grounds that would not also exclude humans.
Second, the problem of trying to program an AI in such a way that your values do not determine the values it acquires, is isomorphic to the problem of trying to write a program to do Bayesian analysis in a way that will not be influenced by your priors; or trying to evaluate a new scientific idea that isn't influenced by your current scientific paradigm. It can't be done, except by defining your terms in a way that hides the problem.
Third, the greatest concern here is how much respect will be given to our free will when setting up the AI governor over us. Given that the people doing the setting up unanimously don't believe in free will, the only logical answer is, Zero.
I've spoken with many of the people involved, and haven't heard any of them express views consistent with this - they want their values to be preserved, in fact two said explicitly that they did not care what happened to the universe if their personal values were not preserved - and yet they also believe that their values are extreme minority views among humanity.
Could you add more details? I'm really interested in this issue since I have similar concerns regarding the whole CEV-idea: It presumes that the values of almost every human somehow converge ...
The product of Less Wrong is truth. However, there seems to be a reluctance of the personality types here - myself included - to sell that product. Here's my evidence:
We actually label many highly effective persuasive strategies that can be used to market our true ideas as "dark arts". What's the justification for this negative branding? A necessary evil is not evil. Even if - and this is a huge if - our future utopia is free of dark arts, that's not the world we live in today. Choosing not to use them is analogous to a peacenik wanting to rid the world of violence by suggesting that police not use weapons.
We treat our dislike of dark arts as if it's a simple corollary of the axiom of the virtue of truth. Does this mean we assume the ends (more people believe the truth) doesn't justify the means (persuasion to the truth via exploiting cognitive biases)? Or are we just worried about being hypocrites? Whatever the reason, such an impactful assumption deserves an explanation. Speaking practically, the successful practice of dark arts requires the psychological skill of switching hats, to use Edward de Bono's terminology. While posting on Less Wrong, we can avoid and are in fact praised for avoiding dark arts, but we need to switch up in other environments, and that's difficult. Frankly, we're not great at it, and it's very tempting to externalize the problem and say "the art is bad" rather than "we're bad at the art".
Our distaste for rhetorical tactics, both aesthetically and morally, profoundly affects the way we communicate. That distaste is tightly coupled with the mental habit of always interpreting the value of what is said purely for its informational content, logical consistency, and insight. I'm basing the following question on my own introspection, but I wonder if this almost religiously entrenched mental habit could make us blind to the value of the art of persuasion? Let's imagine for a moment, the most convincing paragraph ever written. It was truly a world-wonder of persuasion - it converted fundamentalist Christians into atheists, suicide bombers into diplomats, and Ann Coulter-4-President supporters into Less Wrong sycophants. What would your reaction to the paragraph be? Would you "up-vote" this work of genius? No way. We'd be competing to tell the fundamentalist Christian that there were at least three argument fallacies in the first sentence, we'd explain to the suicide bomber that the rhetoric could be used equally well to justify blowing us all up right now, and for completeness we'd give the Ann Coulter supporter a brief overview of Bayesianism.