Why is Bayes' Rule useful? Most explanations of Bayes explain the how of Bayes: they take a well-posed mathematical problem and convert given numbers to desired numbers. While Bayes is useful for calculating hard-to-estimate numbers from easy-to-estimate numbers, the quantitative use of Bayes requires the qualitative use of Bayes, which is noticing that such a problem exists. When you have a hard-to-estimate number that you could figure out from easy-to-estimate numbers, then you want to use Bayes. This mental process of testing beliefs and searching for easy experiments is the heart of practical Bayesian thinking. As an example, let us examine 1 Kings 3:16-28:
Now two prostitutes came to the king and stood before him. One of them said, “Pardon me, my lord. This woman and I live in the same house, and I had a baby while she was there with me. The third day after my child was born, this woman also had a baby. We were alone; there was no one in the house but the two of us.
“During the night this woman’s son died because she lay on him. So she got up in the middle of the night and took my son from my side while I your servant was asleep. She put him by her breast and put her dead son by my breast. The next morning, I got up to nurse my son—and he was dead! But when I looked at him closely in the morning light, I saw that it wasn’t the son I had borne."
The other woman said, “No! The living one is my son; the dead one is yours.”
But the first one insisted, “No! The dead one is yours; the living one is mine.” And so they argued before the king.
The king said, “This one says, ‘My son is alive and your son is dead,’ while that one says, ‘No! Your son is dead and mine is alive.’”
Notice that Solomon explicitly identified competing hypotheses, raising them to the level of conscious attention. When each hypothesis has a personal advocate, this is easy, but it is no less important when considering other uncertainties. Often, a problem looks clearer when you branch an uncertain variable on its possible values, even if it is as simple as saying "This is either true or not true."
Then the king said, “Bring me a sword.” So they brought a sword for the king. He then gave an order: “Cut the living child in two and give half to one and half to the other.”
The woman whose son was alive was deeply moved out of love for her son and said to the king, “Please, my lord, give her the living baby! Don’t kill him!”
But the other said, “Neither I nor you shall have him. Cut him in two!”
Then the king gave his ruling: “Give the living baby to the first woman. Do not kill him; she is his mother.”
Solomon considers the empirical consequences of the competing hypotheses, searching for a test which will favor one hypothesis over another. When considering one hypothesis alone, it is easy to find tests which are likely if that hypothesis is true. The true mother is likely to say the child is hers; the true mother is likely to be passionate about the issue. But that's not enough; we need to also estimate how likely those results are if the hypothesis is false. The false mother is equally likely to say the child is hers, and could generate equal passion. We need a test whose results significantly depend on which hypothesis is actually true.
Witnesses or DNA tests would be more likely to support the true mother than the false mother, but they aren't available. Solomon realizes that the claimant's motivations are different, and thus putting the child in danger may cause the true mother and false mother to act differently. The test works, generates a large likelihood ratio, and now his posterior firmly favors the first claimant as the true mother.
When all Israel heard the verdict the king had given, they held the king in awe, because they saw that he had wisdom from God to administer justice.
Sure, that registers -- if there were a thriving microbial ecosystem on Mars, I'd consider it immoral to wipe it out utterly simply for the sake of one human being. Though I think my function-per-individual is more complicated than that; wiping it out because that one human is a hypochondriac is more-wrong in my perception than wiping it out because, let's say, that one human is an astronaut stranded in some sort of weird microbial mat, and the only way to release them before they die is to let loose an earthly extremophile which will, as a consequence, propagate across Mars and destroy all remaining holdouts of the local biosphere. That latter is very much more a tossup, such that I don't view other humans going 'Duh, save the human!' as exactly committing an atrocity or compounded the wrong. Sometimes reality just presents you with situations that are not ideal, or where there is no good choice. No-win situations happen, unsatisfying resolutions and all. That doesn't mean do nothing; it just means trying to set up my ethical and moral framework to make it impossible feels silly.
To be honest, that's all this debate really seems to be to me -- where do we set that fence? And I'm convinced that the decision point is more cultural and personal than anything, such that the resulting discussion does not usefully generalize.
And once I do, even if my decision was as rational as it can be under the circumstances and I've identified a set of priorities most folks would applaud in principle, there's still the potential for regrets and no-win situations. While a moral system that genuinely solved that problem would please me greatly, I see no sign that you've stumbled upon it here.