Before certain MIRI papers, I came up with a steelman in which transparently written AI could never happen due to logical impossibility. After all, humans do not seem transparently written. One could imagine that the complexity necessary to approximate "intelligence" grows much faster than the intelligence's ability to grasp complexity - at least if we mean the kind of understanding that would let you improve yourself with high probability.
This scenario seemed unlikely even at the time, and less likely now that MIRI's proven some counterexamples to closely related claims.
I'm not sure I understand the logic of your argument. I suspect I do not understand what you mean by transparently written.
If Strong AI turns out to not be possible, what are our best expectations today as to why?
I'm thinking of trying myself at writing a sci-fi story, do you think exploring this idea has positive utility? I'm not sure myself: it looks like the idea that intelligence explosion is a possibility could use more public exposure, as it is.
I wanted to include a popular meme image macro here, but decided against it. I can't help it: every time I think "what if", I think of this guy.