If you're going to shut up and calculate, you need to calculate not only the benefit and harm from the experiment, you need to calculate the benefit and harm from weakening the ethics code. You can't weaken ethics just for one experiment; if you weaken it for experiments where the benefit outweighs the harm, you'll also weaken it for experiments where the harm outweighs the benefit but the people performing the experiment value harm to some people more than to others, and for experiments where the experimenters just don't know how to calculate, and for experimenters where an unspoken goal of the experiment is to actually cause harm, and....
It's like asking if we should allow courts to use evidence gained from an illegal search. If you "shut up and calculate" the costs of freeing a criminal versus finding a criminal guilty, you'll determine that it's always good to use the evidence. But if you allow the use of evidence gained from illegal searches, you'll incentivize illegal searches, and that incentivization must be included in any calculation.
I could even phrase this in the language of precommitting: you precommit to always following the ethics code because following an ethics code is, on the average, advantageous but you don't get the advantages unless you're the type of person who's willing to follow it even in disadvantageous situations.
...If you're going to shut up and calculate, you need to calculate not only the benefit and harm from the experiment, you need to calculate the benefit and harm from weakening the ethics code. You can't weaken ethics just for one experiment; if you weaken it for experiments where the benefit outweighs the harm, you'll also weaken it for experiments where the harm outweighs the benefit but the people performing the experiment value harm to some people more than to others, and for experiments where the experimenters just don't know how to calculate, and for ex
A look at all natural foods through the lenses of Bayesianism, optimisation, and friendly utility functions.
How should we consider foods that claim to be "all natural"? Or, since that claim is a cheap signal, foods that have few ingredients, all of them easy to recognise and all "natural"? Or "GM free"?
From the logical point of view, the case is clear: valuing these foods is nothing more that the appeal to nature fallacy. Natural products include many pernicious things (such as tobacco, hemlock, belladonna, countless parasites, etc...). And the difference between natural and not-natural isn't obvious: synthetic vitamin C is identical to the "natural" molecule, and gene modifications are just advanced forms of selective breeding.
But we're not just logicians, we're Bayesians. So let's make a few prior assumptions:
Now let's see the food industry as optimising along a few axis:
The last point is the weakest one, though. Long term health impacts are fiendishly hard to measure (by anyone), and consumers' memories are short. The main things that the food industry wants to avoid are blaring headlines like "salmon saturated with mercury", "sugar linked to cancer", or "food industry coverup". Even there, journalists can create scandals out of very little (or ignore major ones if something more juicy comes along), so the pressure to actually prioritise long term consumer health is very weak. So most optimisation pressure is along the first two axis.
So the food industry will strongly push to decrease cost and increase satisfaction, while mostly taking a random walk on long term consumer health. Given the assumptions above, this means that we'd expect the long term health impacts to worsen (because there are far more negative products than good one). This somewhat similar to the importance of programming everything of value into a friendly AI, lest the things not programmed get squeezed out.
Now, what does "all natural" mean in this setting? It means that the natural food industry is facing a massively constrained optimisation problem. They are extremely limited in what they can do (compared with the rest of the industry), and it mostly involves shuffling around with products we suspect to be benign or positive. Similarly with GM modifications: selective breeding is much slower and uncertain, so the optimisation pressure is less: they literally can't change things as much or as fast. One supporting argument for this is that all natural or GM-free products tend to be more expensive or less satisfying than others, demonstrating less optimisation pressure.
This is entirely not a result I expected to find. Pushing it to the extreme, it would seem that the most traditional and unchanged food (after removing stuff we know to be bad) is likely to be best, as long as people don't get too inventive with them.