Roko, not all minds are good optimizers, not everything with a mind has everything within it proceeding according to optimization, and humans in particular are hacks. I think it should be obvious that I do not regard humans as expected utility maximizers, all bloggings considered, and I've written before about structurally difficult Friendly AI problems that are hard to interpret in terms of EU maximizing (e.g. "preserve individual self-determination"). Still, the insight is the insight.

Come now. It's fine to realise you've made a mistake. But in itself this does not make you as smart as a Protector.

If I'm as smart as Larry Niven, I'm as smart as a Protector. Vinge's Law: No character can be realistically depicted as being qualitatively smarter than the author. Next you'll be telling me that I'm not as smart as that over-enthusiastic child, Kimball Kinnison.