This post is interesting but it does not mention markets, and I argue that markets are intelligent. Not merely a team, or a company, or a civilization. I'm specifically talking about a collection of entities who transact with each other and behave in their own self interest. A market is an organizational technology that allows greater intelligence than could otherwise be achieved.
I agree that markets are themselves mildly intelligent, but I think you are underestimating how weak they are compared to what can exist. eg, imagine a market between 15 housecats (say, imagine they all live nearby behind walls, and they can trade things between each other by dropping them in a chute, if the other cat puts in enough payment treats, or something). anyway, cats each have about 10 trillion synapses in 760 million neurons, vs a single human with about 150 trillion synapses in 100 billion neurons. the market is an enormously limiting information bottleneck compared to a higher-bandwidth communication system, and the adversarial nature makes it hard to transmit information usefully. There's no way the cats could implement market mechanisms well enough to make use of their neurons to, eg, learn abstract math efficiently. The human can. sure, the smartest cat will end up with the most stuff, and the market will have thereby allowed the most intelligent cat to direct the system. If some cats are better at some things, gains from trade will occur and they can each specialize somehow. Maybe one hunts their enclosure and the other maps the area or something, I'm not really imagining this in much detail. But even then, competitive low bandwidth communication just doesn't compare favorably to an integrated brain.
AI would be able to beat us because of being a big integrated thing that communicates far better than we can. In the short term it manipulates us like tiktok and youtube (or more accurately, the market - an intelligent, misaligned system - rewards humans who implement AI to manipulate other humans, which is, you know, why tiktok and youtube are big sites), then in the long term it has no need of us because the robots can make more robots and the ai is better at designing AI, not currently true but visibly possible now.
6mesaoptimizer
Please read Nick Bostrom's "Superintelligence", it would really help you understand where everyone here has in mind when they talk about AI takeover.
This post is interesting but it does not mention markets, and I argue that markets are intelligent. Not merely a team, or a company, or a civilization. I'm specifically talking about a collection of entities who transact with each other and behave in their own self interest. A market is an organizational technology that allows greater intelligence than could otherwise be achieved.