The point being that actually, it is worthwhile to point out simply that it is not a logical necessity - because people actually believe that. Once that is accepted, it clears the way for discussion of the actual probability that the AI does such a good job.

Therefore there is not one thing wrong with the excerpt that you quoted (and if you have a problem with another part of the essay, you should at least point out where the fallacy is).

To address the question of the likelihood of the AI patching things up itself:

How much trust do we put in human programmers? In one instance, they would have to create a dynamic that can apply transformations to Nobel laureates; in the other, they must create a dynamic that can apply transformations to a massive number of mutually antagonistic, primitive, low-IQ and superstitious minds.

Furthermore, although speculation about the details of the implementation becomes necessary, using a small group of minds the programmers could learn about these minds in vast detail, specifically identifying any particular problems and conducting tests and trials, whereas with 7 billion or more minds this is impossible.

The initial dynamic is supposed to be capable of generating an improved dynamic. On the other hand, there are certain things the AI can’t help with. The AI does have vast knowledge of its own, but the programmers have specified the way in which the AI is to “increase knowledge” and so forth of the humans in the first place. This is the distinction wedrifid seems to have missed. If this specification is lousy in the first place, then the output that the AI extracts from extrapolating the volition of humanity might be some way off the mark, in comparison to the ouput if “increasing knowledge” etc. was done in an ideal fashion.

The AI may then go on to implement a new CEV dynamic – but this might be a lousy equilibrium generated by an original poor implementation of transforming the volition of humanity, and this poor reflection of human volition is down to the abilities of the human programmers.

On the other hand, it might take a suboptimal initial dynamic (with suboptimal specifications of “increase knowledge”, “grow up closer together etc.), and manage to locate the ideal dynamic. What I dispute is that this is “blatantly” obvious. That is (motivated) overconfidence regarding a scenario that is purely theoretical, and very vague at this point.

And I certainly dispute that it is necessary “by definition”, which is all I actually claimed in my essay!!

In other words, a superintelligence is not immune to GIGO. Getting an output of some kind from the CEV does not guarantee that the superintelligence has circumvented this problem.

Edit: He has disappeared. How is this for a rational quote:

"This is always the tactic of the denialist: lie and run. He never stays for a fight; he never admits error, even the most glaring; his goal is to pack the maximum insult into the minimum number of words."