Magnus Carlsen is closer in ELO to Stockfish than median human.

Chess is a bad example.

Here's a useful rule of thumb: Every 100 Elo is supposed to give you a 30% edge. Or play around with this: https://wismuth.com/elo/calculator.html

This means that if a 1400 plays a 1500, the 1500 should win about 30% more than the 1400. Totally normal thing that happens all the time.

It also means that if a one-million Elo AI plays a one-million-one-hundred Elo AI, the one-million-one-hundred should win 30% more than the one-million. This is completely absurd, because actual superintelligences are just going to draw each other 100% of the time. Ergo, there can never be a one-million Elo chess engine.

It's like chess has a ceiling, where as you get close to that ceiling all the games become draws and you can't rise further. The ceiling is where all the superintelligences play, but the location of the ceiling is just a function of the rules of chess, not a function of how smart the superintelligences are. Magnus Carlsen is closer to the ceiling than he is to the median human's level, which can be taken as merely a statement about how good he is at chess relative to its rules.

In the game "reality," there's probably still a ceiling, but that ceiling is so high that we don't expect any AIs that haven't turned the Earth into computronium to be anywhere near it.

I agree with most of these claims.  However, I disagree about the level of intelligence required to take over the world, which makes me overall much more scared of AI/doomy than it seems like you are.  I think there is at least a 20% chance that a superintelligence with +12 SD capabilities across all relevant domains (esp. planning and social manipulation) could take over the world.  

I think human history provides mixed evidence for the ability of such agents to take over the world.  While almost every human in history has failed to accumulate massive amounts of power, relatively few have tried.  Moreover, when people have succeeded at quickly accumulating lots of power/taking over societies, they often did so with surprisingly small strategic advantages.  See e. g. this post; I think that an AI that was both +12 SD at planning/general intelligence and social manipulation could, like the conquistadors, achieve a decisive strategic advantage without having to have some kind of crazy OP military technology/direct force advantage.  Consider also Hitler's rise to power and the French Revolution as cases where one actor/a small group of actors was able to surprisingly rapidly take over a country. 

While these examples provide some evidence in favor of it being easier than expected to take over the world, overall, I would not be too scared of a +12 SD human taking over the world.  However, I think that the AI would have some major advantages over an equivalently capable human.  Most importantly, the AI could download itself onto other computers.  This seems like a massive advantage, allowing the AI to do basically everything much faster and more effectively.  While individually extremely capable humans would probably greatly struggle to achieve a decisive strategic advantage, large groups of extremely intelligent, motivated, and competent humans seem obviously much scarier.  Moreover, as compared to an equivalently sized group of equivalently capable humans, a group of AIs sharing their source code would be able to coordinate among themselves far better, making them even more capable than the humans.

Finally, it is much easier for AIs to self modify/self improve than it is for humans to do so.  While I am skeptical of foom for the same reasons you are, I suspect that over a period of years, a group of AIs could accumulate enough financial and other resources that they could translate these resources into significant cognitive improvements, if only by acquiring more compute.  

While the AI has the disadvantage relative to an equivalently capable human of not immediately having access to a direct way to affect the "external" world, I think this is much less important than the AIs advantages in self replication, coordination, an self improvement.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

I'm sceptical of this. Can you say more about why you think this is true?

Assuming a Gaussian distribution, +6 SD is much rarer than +3 SD, which is already quite rare. There's probably less than 10 +6 SD people alive on the earth today, wheras there are ~10 million +3 SD people. Given the role of things like luck, ambition, practical knowledge, etc., it's not surprising that we see several of the +3 SD people accomplishing things far greater than any of the +6 SD g-factor people, purely on the basis of their much greater abundance.

And that's ignoring potential trade-off effects. Among humans, increased intelligence often seems to come at the cost of lowered social skills and practical nature- there are certainly many intelligent people who are good at sociality and practicality, but there is an inverse correlation (though of course, being intelligent also helps directly to make up for those shortcomings). There's no reason to expect that these same trade-offs will be present in an artificial system, who take completely different physical forms, both in size / form-factor, and in the materials and architectures used to build them. And the incentive gradients that govern the development and construction of artificial systems are also quite different from those that shape humans.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

The smartest humans might be more likely to be mentally ill, the smartest monkeys may be more likely to be mentally ill. 

There is no reason to think human intelligence is normally distributed. But if we go with that assumption, +3 SD is 1 in a thousand. Fairly common in intellectual circles. +6 is 1 in a billion. There should be 8 people worldwide that are +6 SD. Even if you knew who those people were (it isn't like the world has a standardized global g factor test that is applied to all humans), you would still have only 8 datapoints, leaving a lot of room for chance. Subjective impressiveness is a subjective measure. 

Discovering something like relitivity takes around +3SD, plus a fair bit of "luck". Some of that luck is being born in a position where you get maths lessons and time to think. Some is deciding to focus your intelligence on physics. Some is all over the place. Like whether or not you read a particular textbook, whether your thinking style is more geometric or formal symbolic. Whether an apple falls on your head. Neurochemical noise. 

When general relativity was discovered, there was probably 1 +6SD human, (smaller population) And statistically they were a peasant farmer never taught anything beyond arthritic, who may well have been conscripted into some war and shot at. Saying "this +6 human failed to invent relativity" is putting an extremely weak upper bound on the capabilities of +6 SD humans. 

If we go into the logic of evolution, selection pressure applies mostly near median. +3SD is already getting to the region where selection pressure is minimal.  Anything beyond that is just getting lucky on the genetic dice. Consider a model where there are 650 genes, each 99% likely to have the correct version, and with a 1% chance of having a deleterious mutation. A +3SD human has all the genes correct. Thus the only way to go beyond that is to not only have all the genes correct, but for randomness to produce a new gene that's even better. The average human has 6.5 bad genes, so in this model, you would need 6.5 beneficial mutations (and no deleterious ones) to get the +6SD human. This is far too unlikely to ever happen.

So without hard data, we can't put reasonable upper bounds on what a +6SD human is capable of. Even with any data you might gather, I don't think it would be easy to learn much about the limits of intelligence. Any signal is mostly about the upper tail produced by evolution. And possibly about who is best at goodhearting your g test. 

Fascinating insight with the chimpanzee. I believe this connects to the argument that is sometimes made: an ASI takeover is unlikely because how widely distributed the physical levers of power are in the real world. Ok... but chimpanzees don't have access to massively scaleable industrial weapon technology, such as nuclear, biological etc. They don't live in an increasingly electronically connected world. They don't rely on an electricity grid, among others, for their many daily needs. Also they live in groups or troops - not tribes, but that was more for the pedantic kick of it.

I think it's a mistake to think of current chess or go engines as being at maximum capability. If we would throw a few billion dollar worth of compute at them they would likely get significantly better. 

Narrow Optimisers Outperform General Optimisers on Narrow Domains

That's true sometimes but not always. Notably, GATO is better at controlling a Sawyer arm than more specialized optimizers. Given that the company that sells the Sawyer arm spent a lot of time developing software to control it, that's impressive. 

Also - I don't really get "the general intelligence is composite anyway" argument. Ok - I also believe that it is. But what would prevent an ASI from being developped as a well-coordinated set of many narrow optimizers?

Also - why the fixation on 12 SD? It's not that high really. It sounds high to a human evaluating another human. Bostrom made a good point on this - the need to step out of the antropomorphic scale. This thing could very well reach 120 SD (the fact that we wouldn't even know how to measure and recognize 120 SD is just an indication of our own limitations, nothing more), and make every human look like a clam.