Do you really have that preference?

For example, if every but one of trillions of humans was being tortured and had dust specks, would you feel like trading the torture-free human's freedom from torture for the removal of specks from the tortured. If that were so, then you just are showing a fairly usual preference (inequality is bad!) which is probably fine as an approximation of stuff you could formalize consequentially.

But that's just an example. Often there's some context in which your moral intuition is reversed, which is a useful probe.

(usual caveat: haven't read the sequences)

Topic for discussion: Less Wrongians are frequentists to a greater extent than most folk who are intuitively Bayesian. The phrase "I must update on" is half code for (p<0.05) and half signalling, since presumably you're "updating" a lot, just like regular humanssssssssssssssssssssssssssssssssss.