evolution hit on some necessary extraordinarily unlikely combination to give us intelligence and for P vs NP reasons we can't find it
For this one, you also need to explain why we can't reverse-engineer it from the human brain.
"Reverse-engineer" is an almost perfect metaphor for "solve an NP problem."
This is not true at all.
"Solve an NP problem" is "you are looking for a needle in a haystack, but you will know when you find it."
"Reverse engineer" is "there is a machine that seems to find needles in haystacks quickly. It has loops of copper wire, and plugs into a wall socket. Can you copy it and build another one?"
It just seems to me that if you are trying to reverse engineer a complicated object of size O(k) bits (which can be a hard problem if k is large, as is the case for a complicated piece of code or the human brain), then the search problem where the object is the solution must have been exponential in k, and so is much much worse.
If Strong AI turns out to not be possible, what are our best expectations today as to why?
I'm thinking of trying myself at writing a sci-fi story, do you think exploring this idea has positive utility? I'm not sure myself: it looks like the idea that intelligence explosion is a possibility could use more public exposure, as it is.
I wanted to include a popular meme image macro here, but decided against it. I can't help it: every time I think "what if", I think of this guy.