You might enjoy this story I wrote a few months ago, also about AI doom and also set in the future: https://www.lesswrong.com/posts/BgTsxMq5bgzKTLsLA/this-is-already-your-second-chance

Self-review: It's been long enough that I've forgotten most of the details of the post, so it's a good time to re-read it and get a sense of what it reads like for someone who's discovering the content for the first time. I still believe most of the ideas here are correct. My goal here was to write a bottom-up overview of how the basic molecular structure of DNA might "inevitably" lead to stuff like sexually-dimorphic ornaments, after a long chain of events. The point I wanted to get at was that the male/female binary, which human cultures often depict as a metaphysical fundamental principle of the universe, results from a series of pretty prosaic evolutionary constraints. Love itself isn't quite as magical and central to the Universe as we make it out to be. This post hints at the idea, but I could have expanded more in that direction. In hindsight, I think most people saw this as a kind of textbook explanation of the evolution of sex, and I guess this post does a OK job at this. However, it's completely skipping over some important theories, namely the Red Queen Race and the handicap principle. Maybe I should add some links at the end. (To my defense, I did include a disclaimer about this not being a complete overview of the field.)

Which one? I hope it's not the one where you have to put chocolate, because this is the most crucial instruction.

it's no biology lab

I'm afraid you're overestimating how well biologists follow the safety procedures. I wouldn't be surprised if we all had fluorescent bacteria in our guts.

Oh, that's a really good point. Actually, it might be common for chemists to work with panels of related molecules, while in clinical trials they only work with one purified drug candidate. This makes it less likely for them to discover things by accident. Surely a piece of the puzzle!

Sure, all these stories totally sound like urban legends, but the sweeteners are out there and I don't see how they could have been discovered otherwise (unless they were covertly screening drugs on a large number of people).

That's a great question, this is totally mysterious to me. There are a lot of examples of people putting thaumatin in transgenic fruits or vegetables (and somehow in the milk of transgenic mice because there's always one creepy study), but I don't know why it hasn't been commercialized. It sounds like superfruits would make a nice healthy alternative to palm-oil-and-chocolate-based comfort foods. Maybe it's a regulatory problem?

That sounds exciting! I hadn't seen Elisabeth's comment, I just wrote a reply. Do you think there are modifications I should make to the main text to clarify?

That sounds plausible, but I've not looked into the empirical research on that topic so I can't tell you much more! 

(Sorry I missed your comment)

Here by "reproduce" I just meant "make more copies of itself" in an immediate sense (so reproductive fitness is just "how fast it replicates right now"). For example, in Lenski's long-term evolution experiment, some variants were selected not because they increased the bacteria's daily growth rate, but because they made it easier to acquire further variants that themselves increased the daily growth rate. These "potentiating" variants were initially detrimental (the copy number of these variants decreased in the population), and only after a long long time they took over the population. So, according the definition of reproductive fitness I used, they lead to a lower reproductive fitness – the reason they were eventually selected for is not that they're good for reproduction, but that they're good for evolvability. Of course, you can say that eventually they increased in copy number, but that would be defining "reproduction" in a different way, that I find less intuitive.

Now, is that other definition (how gene copy number increases over the long term) what evolution ultimately selects for? I'm not sure. To quote Kokko's review on the stagnation paradox:

"Trees compete for sunlight and attempt to outshade each other, but when each tree consequently invests in woody growth, the entire forest must spend energy in stem forming and—assuming time or energy trade-offs—will be slower at converting sunlight into seeds than a low mat of vegetation would have been able to. Every individual has to invest in outcompeting others, but the population as a whole is negligibly closer to the light source (the number of photons arriving in the area is still the same). This is why in agriculture, externally imposed group selection to create shorter crops has improved yields."

She gives other examples. In these cases, the number of individuals tend to decrease over time, even in the long run.