Haha! You're right, I spoke too soon. The graph Katja used is the exact same graph used to explain Berkson's paradox and the pattern match hit me so hard I couldn't resist commenting. Lesson learned: think about it for 5 minutes before commenting.
Katja's phenomenon (KP) is not an instance of Berkson's paradox (BP) but I can see how they would often go together. Imagine that you go down a street and pick up everyone who is particularly tastefully dressed. In this group rich people will be overrepresented simply because they more easily afford good clothes. You could naively conclude from this that wealth correlates with taste. This is KP.
THEN you administer some rigorous test of good taste to everyone in the group (whatever that means). And then you discover that now the rich people in you population are *under*performing relative to the poor. For the simple reason that in your population everyone is either rich or has good taste - all the poor ones have good tastes but not all of the rich do. From this you could naively conclude that wealth *negatively* correlates with taste. This is BP.
How would this work in real life? Maybe a casual observer is more likely to fall to KP and think rich <-> good taste while a fashion expert who works exclusively with well-dressed people will think the opposite?
It's called Berkson's paradox and it can be used to explain all kinds of real life observations like "why are all the handsome men I date such jerks" (http://www.slate.com/blogs/how_not_to_be_wrong/2014/06/03/berkson_s_fallacy_why_are_handsome_men_such_jerks.html) or why google discovered that being good at programming competitions negatively correlated with being good at the job.
I feel like this and many other arguments for AI-skepticism are implicitly assuming AGI that is amazingly dumb and then proving that there is no need to worry about this dumb superintelligence.
Remember the old "AI will never beat humans at every task because there isn't one architecture that is optimal at every task. An AI optimised to play chess won't be great at trading stocks (or whatever) and vice versa"? Well, I'm capable of running a different program on my computer depending on the task at hand. If your AGI can't do the same as a random idiot with a PC, it's not really AGI.
I am emphatically not saying that Robin Hanson has ever made this particular blunder but I think he's making a more subtle one in the same vein.
Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it's creators (i.e. actual AGI) and turns its powers to recursive self-improvement - how can that result in anything but FOOM? Doesn't matter if further improvements require more complexity or less complexity or a different kind of complexity or whatever. If human researchers can do it then AGI can do it faster and better because it scales better, doesn't sleep, doesn't eat and doesn't waste time arguing with people on facebook.
This must have been said a million times already. Is this not obvious? What am I missing?