The story you are referring to is On the Origin of Circuits.
The plucky chip was utilizing only thirty-seven of its one hundred logic gates, and most of them were arranged in a curious collection of feedback loops. Five individual logic cells were functionally disconnected from the rest-- with no pathways that would allow them to influence the output-- yet when the researcher disabled any one of them the chip lost its ability to discriminate the tones. Furthermore, the final program did not work reliably when it was loaded onto other FPGAs of the same type.
This has been repeated many times in different domains where machines are used to design something. The output is usually really hard to understand, whether it be code, mathematical formulas, neural network weights, transistors, etc. Of course reverse engineering code in general is difficult, it may not be any specific problem with GAs.
Furthermore, the final program did not work reliably when it was loaded onto other FPGAs of the same type.
This makes an interesting contrast with biological evolution. The "programs" it comes up with do run quite reliably when loaded onto other organisms of the same type. If fact, the parts of slightly different programs from different individuals can be jumbled together at random and it still works! Often, you can take a component from one organism and insert it into very distantly related one and it still works! On top of that, organisms a...
If Strong AI turns out to not be possible, what are our best expectations today as to why?
I'm thinking of trying myself at writing a sci-fi story, do you think exploring this idea has positive utility? I'm not sure myself: it looks like the idea that intelligence explosion is a possibility could use more public exposure, as it is.
I wanted to include a popular meme image macro here, but decided against it. I can't help it: every time I think "what if", I think of this guy.