Why should we assume that vastly increased intelligence results in vastly increased power?
A common argument I see for intelligence being powerful stems from two types of examples:
Humans are vastly more powerful than animals because humans are more intelligent than animals. Thus, an AGI vastly more intelligent than humans would also have similarly overwhelming power over humans.
X famous person caused Y massive changes in society because of their superior intelligence. Thus, an AGI with even more intelligence would be able to effect even larger changes.
It's possible that there is a ceiling to intelligence gains. It's also possible that there isn't. Looking at the available evidence there doesn't seem to be so - a single ant is a lot less intelligent than a lobster, which is less intelligent than a snake, etc. etc. While it would be nice (in a way) if there was a ceiling, it seems more prudent to assume that there isn't, and prepare for the worst. Especially as by "superintelligent", you shouldn't think of double, or even triple Einstein, rather you should think of a whole other dimension of intelligence, like the difference between you and a hamster.
As to your specific counterarguments:
1. It's both, really. But yes - complex language allows humans to keep and build upon previous knowledge. Humans advantage is in the gigantic amounts of know how that can be passed on to future generations. Which is something that computers are eminently good at - you can keep a local copy of Wikipedia in 20GB.
2. Good point. But it's not just luck. Yes, luck plays a large part, but it's also resources (in a very general sense). If you have the basic required talent and a couple of billions of dollars, I'm pretty sure you could become a hollywood star quite quickly. The point is that a superintelligence won't have a similar level of intelligence to any one else around. Which will allow it to run circles around everyone. Like if a Einstein level intelligence decided to learn to play chess and started playing against 5 year olds - they might win the first couple of games, but after a while you'd probably notice a trend...
3. Intelligence is an advantage. Quite a big one, generally speaking. But in society most people are generally at the same level if you compare them to e.g. Gila monsters (because we're talking about superintelligence). So it shouldn't be all that surprising that other resources are very important. While many powerful people don't seem to be intelligent, they tend to be either cunning (which is a different kin
Why should we assume that vastly increased intelligence results in vastly increased power?
A common argument I see for intelligence being powerful stems from two types of examples:
Howev... (read more)