What does that mean for coming to a reflective equilibrium about ethics?
Are you talking about CEV? Civilization as we know it will end long before people agree about metaethics.
Before CEV, we have to do a rough estimate of our personal extrapolated volition so we know what to do. One way to do this is to extrapolate our volition as far as we can see by, e.g., thinking about ethics.
I intuitively feels that X is good and Y is bad. I believe morality will mostly fit my intuitions. I believe morality will be simple. I know my intuitions, in this case, are pretty stupid. I can't find a simple system that fits my intuitions here. What should I do? How much should I suck up and take the counterintuitiveness? How much should I suck up and take complex morality?
These are difficult questions.
Two or three months ago, my trip to Las Vegas made me ponder the following: If all gambles in the casinos have negative expected values, why do people still engage in gambling - especially my friends fairly well-versed in probability/statistics?
Suffice it to say, I still have not answered that question.
On the other hand, this did lead me to ponder more about whether rational behavior always involves making choices with the highest expected (or positive) value - call this Rationality-Expectation (R-E) hypothesis.
Here I'd like to offer some counterexamples that show R-E is clearly false, to me at least. (In hindsight, these look fairly trivial but some commentators on this site speak as if maximizing expectation is somehow constitutive of rational decision making - as I used to. So, it may be interesting for those people at the very least.)
A is a gamble that shows that choices with negative expectation can sometimes lead to net pay off.
B is a gamble that shows that choices with positive expectation can sometimes lead to net costs.
As I'm sure you've all noticed, expectation is only meaningful in decision-making when the number of trials in question can be large (or more precisely, large enough relative to the variance of the random variable in question). This, I think, in essence is another way of looking at Weak Law of Large Numbers.
In general, most (all? few?) statistical concepts make sense only when we have trials numerous enough relative to the variance of the quantities in question.
This makes me ponder a deeper question, nonetheless.
Does it make sense to speak of probabilities only when you have numerous enough trials? Can we speak of probabilities for singular, non-repeating events?