@Bambi: Your observers are not anywhere remotely near imaginative enough to suppose something like a human, having never seen a human and having "no reason to believe anything like that is possible". They would simply expect evolution to continue the way it always has. In hindsight, whatever happens in AI, whatever it may be, will seem just as inevitable as the way you're making it sound; the trick is seeing it in advance, for which purpose the future is always absurd.

@Hanson: Do you seriously put forth such a view, or are you playing Devil's Advocate? It seems to me that in my chosen challenge of reflectivity there are basic things that I do not understand, and while it is always extremely dangerous to say this sort of thing, I don't know of anyone else who does understand it. The notion that we have collected all major insights needed for AI sounds very odd to me; people just don't seem that unconfused.