You are assuming that the programmer's personal desired reflect what is best for humans as whole.
[...]
if I could program a super-smart AI to push through my personal notion of "good" I wouldn't want to because I'd rather let collective decision making occur than impose my view on everyone.
But my point is, if thats what you want, then it will do it. If you want to make it a democracy, then you can spend years trying to figure out every possible exception and end up with a disaster like whats presented in this post, or you can make the AI and it will organize everything the way you want it as best it can without creating any bizzare loopholes that could destroy the world. Its always going to be a win-win for whoever created it.
This is aside from other issues like the fact that there likely won't be a single programmer for such an AI but rather a host of people working on it.
Possibly, though I doubt it. But even if it is, you can just do that democracy thing on the group in question, not the whole world. Also, until your AI is smart enough and powerful enough to work at that level, its going to be extremely dangerous to declare that the AI will be in charge of the world from then on. Even if its working perfectly, without the proper resources and strategy in place, its going to be very though to just "take over" and it will likely cost lives. Infact, to me thats the scariest part of AI. Good or bad, at some point the old system is going to have to be abolished.
A lot of these issues are discussed in much more detail in the sequences and older posts. You might be downvoted less if you read more of those instead of rehashing issues that have been discussed previously. At least if you read those, you'll know what arguments have been made before and which have not been brought up. Many online communities one can easily jump into without reading much of their recommended reading. Unfortunately, that's not the case for Less Wrong.
I only have so much time in a day and in that time there is only so much I can read/do. But I do try.
Its always going to be a win-win for whoever created it.
Well, thankfully a lot of the people here care enough about the opinions of others that they want to work out a framework that will work well for others. Note incidentally, that it isn't necessarily the case that it will even be a win for the programmer. Bad AI's can end up trying to paperclip the Earth . Even the democracy example would be difficult for the AI to achieve. Say for example that I tell the AI to determine things with a democratic system and to give that a highest priority and then a ...
It’s the year 2045, and Dr. Evil and the Singularity Institute have been in a long and grueling race to be the first to achieve machine intelligence, thereby controlling the course of the Singularity and the fate of the universe. Unfortunately for Dr. Evil, SIAI is ahead in the game. Its Friendly AI is undergoing final testing, and Coherent Extrapolated Volition is scheduled to begin in a week. Dr. Evil learns of this news, but there’s not much he can do, or so it seems. He has succeeded in developing brain scanning and emulation technology, but the emulation speed is still way too slow to be competitive.
There is no way to catch up with SIAI's superior technology in time, but Dr. Evil suddenly realizes that maybe he doesn’t have to. CEV is supposed to give equal weighting to all of humanity, and surely uploads count as human. If he had enough storage space, he could simply upload himself, and then make a trillion copies of the upload. The rest of humanity would end up with less than 1% weight in CEV. Not perfect, but he could live with that. Unfortunately he only has enough storage for a few hundred uploads. What to do…
Ah ha, compression! A trillion identical copies of an object would compress down to be only a little bit larger than one copy. But would CEV count compressed identical copies to be separate individuals? Maybe, maybe not. To be sure, Dr. Evil gives each copy a unique experience before adding it to the giant compressed archive. Since they still share almost all of the same information, a trillion copies, after compression, just manages to fit inside the available space.
Now Dr. Evil sits back and relaxes. Come next week, the Singularity Institute and rest of humanity are in for a rather rude surprise!