I found myself thinking along similar lines about a year ago when I was faced with a legitimate moral dilemma. Situations which I can view in the abstract, or which I'm distanced from, I can generally apply dispassionate cost-benefit analysis to; but if I actually find myself in a position where I have to make decisions with moral consequences, I'll find myself agonising over what kind of person it makes me.
There's an extra frustrating element to this, because some decisions only have moral consequences as far as "what kind of person they make me", and with enough practice, it's easy to find ignoble motives for any option I might select when given such a decision. This can be quite debilitating, but thankfully I don't face too many moral dilemmas.
Disclaimer: I am not a philosopher, so this post will likely seem amateurish to the subject matter experts.
LW is big on consequentialism, utilitarianism and other quantifiable ethics one can potentially program into a computer to make it provably friendly. However, I posit that most of us intuitively use virtue ethics, and not deontology or consequentialism. In other words, when judging one's actions we intuitively value the person's motivations over the rules they follow or the consequences of said actions. We may reevaluate our judgment later, based on laws and/or actual or expected usefulness, but the initial impulse still remains, even if overridden. To quote Casimir de Montrond, "Mistrust first impulses; they are nearly always good" (the quote is usually misattributed to Talleyrand).
Some examples:
I am not sure how to classify religious fanaticism (or other bigotry), but it seems to require a heavy dose of virtue ethics (feeling righteous), in addition to following the (deontological) tenets of whichever belief, with some consequentialism (for the greater good) mixed in.
When I try to introspect my own moral decisions (like whether to tell the truth, or to cheat on a test, or to drive over the speed limit), I can usually find a grain of virtue ethics inside. It might be followed or overridden, sometimes habitually, but it is always there. Can you?