Putting error-correction codes in the genetic code is an interesting idea. In the context of the Pikachu thought experiment, though, here what I think would happen in the long run: because of the drift barrier, evolution can't distinguish between a ~1/N error rate and a zero error rate. So, there's nothing to prevent the rest of the machinery to become less accurate, until the error rate reaches 1/N after error correction. Now that I think about, you could probably keep things in control by systematically sequencing the genes for the replication machinery and breeding based on that. There is a spark of hope.

Oh yeah, I mean to compare things that have the same functionality (e.g. human-made butterfly robot vs natural butterfly). Obviously shovels are more simple than butterflies. But, seeing stuff like the wax motor and other examples people have posted, humans are definitely capable of coming up with great simple mechanisms, and I underestimated that. Thanks for bringing it up.

Kudos for taking the challenge! If I understand correctly, your first point is actually pretty similar to how E. coli follows gradients of nutrients, even when the scale of the gradient is much larger than the size of a cell.

You might enjoy this story I wrote a few months ago, also about AI doom and also set in the future: https://www.lesswrong.com/posts/BgTsxMq5bgzKTLsLA/this-is-already-your-second-chance

Self-review: It's been long enough that I've forgotten most of the details of the post, so it's a good time to re-read it and get a sense of what it reads like for someone who's discovering the content for the first time. I still believe most of the ideas here are correct. My goal here was to write a bottom-up overview of how the basic molecular structure of DNA might "inevitably" lead to stuff like sexually-dimorphic ornaments, after a long chain of events. The point I wanted to get at was that the male/female binary, which human cultures often depict as a metaphysical fundamental principle of the universe, results from a series of pretty prosaic evolutionary constraints. Love itself isn't quite as magical and central to the Universe as we make it out to be. This post hints at the idea, but I could have expanded more in that direction. In hindsight, I think most people saw this as a kind of textbook explanation of the evolution of sex, and I guess this post does a OK job at this. However, it's completely skipping over some important theories, namely the Red Queen Race and the handicap principle. Maybe I should add some links at the end. (To my defense, I did include a disclaimer about this not being a complete overview of the field.)

Which one? I hope it's not the one where you have to put chocolate, because this is the most crucial instruction.

it's no biology lab

I'm afraid you're overestimating how well biologists follow the safety procedures. I wouldn't be surprised if we all had fluorescent bacteria in our guts.

Oh, that's a really good point. Actually, it might be common for chemists to work with panels of related molecules, while in clinical trials they only work with one purified drug candidate. This makes it less likely for them to discover things by accident. Surely a piece of the puzzle!

Sure, all these stories totally sound like urban legends, but the sweeteners are out there and I don't see how they could have been discovered otherwise (unless they were covertly screening drugs on a large number of people).

That's a great question, this is totally mysterious to me. There are a lot of examples of people putting thaumatin in transgenic fruits or vegetables (and somehow in the milk of transgenic mice because there's always one creepy study), but I don't know why it hasn't been commercialized. It sounds like superfruits would make a nice healthy alternative to palm-oil-and-chocolate-based comfort foods. Maybe it's a regulatory problem?