If you are not putting effort into choosing a utility function, building this AGI seems as likely as building an FAI
You've made a lot of good comments in this thread, but I disagree with this. As likely?
It seems you are assuming that every possible point in AI mind space is equally likely, regardless of history, context, or programmer intent. This is like saying that, if someone writes a routine to sort numbers numerically, it's just as likely to sort them phonetically.
It seems likely to me that this belief, that the probability distribution over AI mindspace is flat, has become popular on LessWrong, not because there is any logic to support it, but because it makes the Scary Idea even scarier.
Yes, my predictions of what will happen when you don't put effort into choosing a utility function are inaccurate in the case where you do put effort into choosing a utility function.
This is like saying that, if someone writes a routine to sort numbers numerically, it's just as likely to sort them phonetically.
Well, lets suppose someone wants a routine to sort numbers numerically, but doesn't know how to do this, and tries a bunch of stuff without understanding. Conditional on the programmer miraculously achieving some sort of sorting routine, what sho...
http://www.youtube.com/watch?v=ghIj1mYTef4