Furthermore, the final program did not work reliably when it was loaded onto other FPGAs of the same type.
This makes an interesting contrast with biological evolution. The "programs" it comes up with do run quite reliably when loaded onto other organisms of the same type. If fact, the parts of slightly different programs from different individuals can be jumbled together at random and it still works! Often, you can take a component from one organism and insert it into very distantly related one and it still works! On top of that, organisms are very clearly made of parts with specialised, understandable purposes, unlike what you typically see when you look inside a trained neural network.
How does this happen? Can this level of robustness and understandability be produced in artificially evolved systems?
Well the FPGA is a closer analogy to the environment for the organisms. Organisms were heavily optimized for that specific environment. It would be like if you took a species of fish that only ever lived in a specific lake, and put them into a different lake that had a slightly higher PH, and they weren't able to survive as well.
But I don't disagree with your general point, evolution is surprisingly robust. Geoffrey Hinton has a very interesting theory about this here. That sexual reproduction forces genes to randomly recombine each generation, and so it p...
If Strong AI turns out to not be possible, what are our best expectations today as to why?
I'm thinking of trying myself at writing a sci-fi story, do you think exploring this idea has positive utility? I'm not sure myself: it looks like the idea that intelligence explosion is a possibility could use more public exposure, as it is.
I wanted to include a popular meme image macro here, but decided against it. I can't help it: every time I think "what if", I think of this guy.