Pretty much.
There are some moral theories that sound simple and reasonable in the abstract ("maximize happiness", for example) but in reality do not encompass the full range of human value. There are two possible responses to this; you can either examine the evidence and conclude you missed something, or you can decide your theory is self-evidently true and everyone else must be biased, and bite the bullet
Of course, everyone sometimes is biased, and some bullets should be bitten. But when you start advocating forcible wireheading (or eating babies) you should at least reexamine the evidence.
Eliezer may be right. But I predict he hasn't examined binary personhood ... ever? Recently, at any rate.
OK.
With respect to Eliezer in particular, it would greatly surprise me if your disagreement with him was actually about complexity of value as you seem to suggest here, or about unexamined notions of binary personhood. That said, my preference is to let you have your argument with him with him, rather than trying to have your argument with him with me.
With respect to your general point, I'm all in favor of re-examining evidence when it leads me to unexpected conclusions. But as you say, some bullets should be bitten... sometimes it turns out that habitual...
Why is Bayes' Rule useful? Most explanations of Bayes explain the how of Bayes: they take a well-posed mathematical problem and convert given numbers to desired numbers. While Bayes is useful for calculating hard-to-estimate numbers from easy-to-estimate numbers, the quantitative use of Bayes requires the qualitative use of Bayes, which is noticing that such a problem exists. When you have a hard-to-estimate number that you could figure out from easy-to-estimate numbers, then you want to use Bayes. This mental process of testing beliefs and searching for easy experiments is the heart of practical Bayesian thinking. As an example, let us examine 1 Kings 3:16-28:
Notice that Solomon explicitly identified competing hypotheses, raising them to the level of conscious attention. When each hypothesis has a personal advocate, this is easy, but it is no less important when considering other uncertainties. Often, a problem looks clearer when you branch an uncertain variable on its possible values, even if it is as simple as saying "This is either true or not true."
Solomon considers the empirical consequences of the competing hypotheses, searching for a test which will favor one hypothesis over another. When considering one hypothesis alone, it is easy to find tests which are likely if that hypothesis is true. The true mother is likely to say the child is hers; the true mother is likely to be passionate about the issue. But that's not enough; we need to also estimate how likely those results are if the hypothesis is false. The false mother is equally likely to say the child is hers, and could generate equal passion. We need a test whose results significantly depend on which hypothesis is actually true.
Witnesses or DNA tests would be more likely to support the true mother than the false mother, but they aren't available. Solomon realizes that the claimant's motivations are different, and thus putting the child in danger may cause the true mother and false mother to act differently. The test works, generates a large likelihood ratio, and now his posterior firmly favors the first claimant as the true mother.