Disclaimer: I am not a philosopher, so this post will likely seem amateurish to the subject matter experts.
LW is big on consequentialism, utilitarianism and other quantifiable ethics one can potentially program into a computer to make it provably friendly. However, I posit that most of us intuitively use virtue ethics, and not deontology or consequentialism. In other words, when judging one's actions we intuitively value the person's motivations over the rules they follow or the consequences of said actions. We may reevaluate our judgment later, based on laws and/or actual or expected usefulness, but the initial impulse still remains, even if overridden. To quote Casimir de Montrond, "Mistrust first impulses; they are nearly always good" (the quote is usually misattributed to Talleyrand).
Some examples:
Eliezer in a facebook post linked the article When Doing Good Means You’re Bad, which points out that people taking commission to raise a lot of money for charity are commonly considered less moral than those who raise much less but are not paid to do so ("tainted altruism").
This was brought up at a meetup: a pregnant woman in a dire financial situation who decides to have an abortion because she does not want a burden of raising a baby is judged harsher than a woman in a similar situation whose motivation is to avoid inflicting harsh life on the prospective child.
In real-life trolley problems even the committed utilitarians (like commanders during war time) are likely to hesitate before sacrificing lives to save more.
I am not sure how to classify religious fanaticism (or other bigotry), but it seems to require a heavy dose of virtue ethics (feeling righteous), in addition to following the (deontological) tenets of whichever belief, with some consequentialism (for the greater good) mixed in.
When I try to introspect my own moral decisions (like whether to tell the truth, or to cheat on a test, or to drive over the speed limit), I can usually find a grain of virtue ethics inside. It might be followed or overridden, sometimes habitually, but it is always there. Can you?
This is the best response, I think.
"Thou shall not kill" is actually nothing more that a consequentialist heuristic posing as deontological/virtue ethics.
If Batman kills Joker in lieu of a trial, he is a de facto "good guy" authority setting a precedent for such eye-for-eye behavior throughout all of Gotham. That is a potentially powerful meme given Batman's status and could reasonably lead to a norm of ruthless, draconian law enforcement methods for decades to come. There are meta-consequentialist considerations at play.
Killing Joker means, in some sense, Batman had to agree that Joker's ethics -- killing your enemy to advance your ends -- work.
Of course there are times where killing, stealing, lying are consequentially a net positive, but it is very useful to have deontological norms prohibiting those actions and ascribe virtues to those people who follow the rules. It is, in fact, the best consequentialist policy over time.
2ThrustVectoring
Imagine what the police would imagine if they followed the popular conception of what consequentialism is. That's an expected consequence of police action, so if it's worse than what they're doing now, they won't choose to do it (under a sufficiently savvy model of consequentialism.)
Disclaimer: I am not a philosopher, so this post will likely seem amateurish to the subject matter experts.
LW is big on consequentialism, utilitarianism and other quantifiable ethics one can potentially program into a computer to make it provably friendly. However, I posit that most of us intuitively use virtue ethics, and not deontology or consequentialism. In other words, when judging one's actions we intuitively value the person's motivations over the rules they follow or the consequences of said actions. We may reevaluate our judgment later, based on laws and/or actual or expected usefulness, but the initial impulse still remains, even if overridden. To quote Casimir de Montrond, "Mistrust first impulses; they are nearly always good" (the quote is usually misattributed to Talleyrand).
Some examples:
I am not sure how to classify religious fanaticism (or other bigotry), but it seems to require a heavy dose of virtue ethics (feeling righteous), in addition to following the (deontological) tenets of whichever belief, with some consequentialism (for the greater good) mixed in.
When I try to introspect my own moral decisions (like whether to tell the truth, or to cheat on a test, or to drive over the speed limit), I can usually find a grain of virtue ethics inside. It might be followed or overridden, sometimes habitually, but it is always there. Can you?