What prevents someone from, say, simulating a human brain on a computer, then simulating 1,000,000 human brains on a computer, then linking all their cortices with a high-bandwidth connection so that they effectively operate as a superpowered highly-integrated team?
Perhaps the brains would go crazy in some way. Not necessary emotionally, but for example because such connection would amplify some human biases.
Humans already have a lot of irrationalities, but let's suppose (for the sake of a sci-fi story) that the smartest ones among us already are at the local maximum of rationality. Any change in brain structure would make them less rational. (It's not necessary a global maximum; smarter minds can still be possible, but would need a radically different architecture. And humans are not smart enough to do this successfully.) So any experiments by simulating humans in computers would end up with something less rational than humans.
Also, let's suppose that human brain is very sensitive to some microscopic details, so any simplified simulations are dumb or even unconscious, and atom-by-atom simulations are too slow. This would disallow even "only as smart as a human, but 100 times faster" AIs.
That's a good argument. What you're basically saying is that the design of the human brain occupies a sort of hill in design space that is very hard to climb out of. Now, if the utility function is "Survive as a hunter-gatherer in sub-saharan Africa," that is a very reasonable (heck, a very likely) possibility. But evolution hasn't optimized us for doing stuff like designing algorithms and so forth. If you change the utility function to "Design superintelligence", then the landscape changes, and hills start to look like valleys and so on. What I'm saying is that there's no reason to think that we're even at a local optimum for "design a superintelligence".
If Strong AI turns out to not be possible, what are our best expectations today as to why?
I'm thinking of trying myself at writing a sci-fi story, do you think exploring this idea has positive utility? I'm not sure myself: it looks like the idea that intelligence explosion is a possibility could use more public exposure, as it is.
I wanted to include a popular meme image macro here, but decided against it. I can't help it: every time I think "what if", I think of this guy.