Suppose you or I suddenly woke up with superintelligence, but with our existing goal structure intact (and a desire to be cautious).

Can you show me why a decent person like (I presume) you or I with these new powers would suddenly choose to slaughter the human race as an instrumental goal to accomplishing some other ends?

If CEV (or whatever we're up to at the moment) turns out to be a dud and human values are inexorably inconsistent and mutually conflicting, one possible solution would be for me to kill everyone and try again, perhaps building roughly humanish beings with complex values I can actually satisfy that aren't messed up because they were made by an intelligent designer (me) rather than Azathoth.

But really, the problem is that a superintelligent AI has every chance of being nothing like a human, and although we may try to give it innocuous goals we have to remember that it will do what we tell it to do, and not necessarily what we want it to do.

See this Facing the Intelligence Explosion post, or this Sequence post, or Smarter Than Us chapter 6, or something else that says the same thing.