From Costanza's original thread (entire text):
This is for anyone in the LessWrong community who has made at least some effort to read the sequences and follow along, but is still confused on some point, and is perhaps feeling a bit embarrassed. Here, newbies and not-so-newbies are free to ask very basic but still relevant questions with the understanding that the answers are probably somewhere in the sequences. Similarly, LessWrong tends to presume a rather high threshold for understanding science and technology. Relevant questions in those areas are welcome as well. Anyone who chooses to respond should respectfully guide the questioner to a helpful resource, and questioners should be appropriately grateful. Good faith should be presumed on both sides, unless and until it is shown to be absent. If a questioner is not sure whether a question is relevant, ask it, and also ask if it's relevant.
Meta:
- How often should these be made? I think one every three months is the correct frequency.
- Costanza made the original thread, but I am OpenThreadGuy. I am therefore not only entitled but required to post this in his stead. But I got his permission anyway.
It can mean you value short-term reactions instead of long-term consequences. A better analogy would be flavor: candy tastes delicious, but it's long-term consequences are undesirable. In this case, a flawed morality leads you to conclude that because something registers as 'righteous' (say, slaying all the unbelievers), you should go ahead and do it, without realizing the consequences ("because this made everyone hate us, we have even less ability to slay/convert future infidels")
On another level, one can also realize that values conflict ("I really like the taste of soda, but it makes my stomach upset!") -> ("I really like killing heretics, but isn't murder technically a sin?")
Edit: There's obviously numerous other flaws that can occur (you might not notice that something is "evil" until you've done it and are feeling remorse, to try and more tightly parallel your example). This isn't meant to be comprehensive :)