This assumes that every CPU architecture is suitable for the theoretical AGI, it assumes that it can run on every computational substrate. It also assumes that it can easily acquire more computational substrate or create new one. I do not believe that those assumptions are reasonable economically or by means of social engineering. Without enabling technologies like advanced real-world nanotechnology the AGI won't be able to create new computational substrate without the whole economy of the world supporting it.

Supercomputers like the one to simulate the IBM Blue Brain project cannot simply be replaced by taking control of a few botnets. They use highly optimized architecture that needs for example a memory latency and bandwidth bounded below a certain threshold.

Downvoting wrong comments may be harsh for the person being downvoted, but hopefully in the long run it can encourage better comments, or at least make it easier to find good comments.

There may be some flaws in the karma system or the way it's used by the community, but I don't see any obvious improvements, or any other systems that would obviously work better.

Look at mwaser: he complains a lot about being downvoted, but he also got a lot of feedback for what people found lacking in his post. Yes, a portion of the downvotes he gets may be due to factors unrelated to the quality of his arguments (he repeatedly promotes his own blog, and complains about the downvotes being a proof of community irrationality - both can get under people's skin), which is a bit unfortunate, but not a fatal flaw of the karma system.

Wisdom doesn't focus on a small number of goals -- and needs to look at the longest term if it wishes to achieve a maximal number of goals.

For the purposes of an AI, I would just call this intelligence.

I'd like to draw a distinction that I intend to use quite heavily in the future.

What for?

Another nightmare scenario that is constantly harped upon is the (theoretically super-intelligent) consciousness that shortsightedly optimizes one of its personal goals above all the goals of humanity. In game-theoretic terms, this is trading a positive-sum game of potentially infinite length and value for a relatively modest (in comparative terms) short-term gain. A wisdom won't do this.

There's nothing "shortsighted" about it. So what if there are billions of humans and each has many goals? The superintelligence does not care. So what if, once the superintelligence tiles the universe with the thing of its choice, there's nothing left to be achieved? It does not care. Disvaluing stagnation, caring about others' goals being achieved, etcetera are not things you can expect from an arbitrary mind.

A genuine paperclip maximizer really does want the maximum number of possible paperclips the universe can sustain to exist and continue to exist forever at the expense of anything else. If it's smart enough that it can get whatever it wants without having to compromise with other agents, that's what it will get, and it's not being stupid in so doing. It's very effective. It's just unFriendly.

I don't think I agree with your substantive points, but equating intelligence with cross-domain optimization power sure seems like confusing terminology in that, according to common sense usage, intelligence is distinct from rationality and knowledge, and all three of these things factor into good decision-making, which in turn is probably not the only thing determining cross-domain optimization power, depending on how you construe "cross-domain".

Does every rationalist protagonist come out of the box thinking they're the queen or king?

I dunno, but I've never read one that doesn't.

If you considered yourself able to take over the world (which all ultra-rationalist characters (whether pro- or antagonists) seem to) then actually taking it over would be one of the most rational things you could do.

Arguably then, for the audience of people who agree with the statement, the statement itself is not necessary either.

Arguments work by drawing attention to certain things you already know. The act of drawing attention is not void, it's almost the whole point (more so when there are multiple steps, of course, but it's often one step at a time).

Obviously, the comment is courting the undecided. Obviously, many humans are swayed by sheer numbers of people who believe certain things. But that behavior is not rational.

Making conclusions from comments' rating would be largely misguided. On the other hand, paying attention depending on comments' rating is a necessary evil with limited biasing effects.

Of course it's a bad argument when considered as directed to you

Prejudicial strawman. I never said that it was a bad argument. I never said anything close.

Hmm, do you mean that you agree with the statement? Something else? I don't understand.

"This premise is VERY flawed"

This is a statement that can be made about any premise. It is backed by no supporting evidence.

A statement needs to be backed by arguments if that's necessary to make people agree with it. Some statements people already agree with, in which case no supporting arguments are necessary. Not every premise is such that people already agree with a statement that it's "very flawed".

Of course it's a bad argument when considered as directed to you (because it fails to move your beliefs), but a good one when seen as directed to other readers.