The argument that I was making or, maybe, just implying is a version of the argument for deontological ethics. It rests on two lemmas: (1) You will make mistakes; (2) No one is a villain in his own story.

To unroll a bit, people who do large-scale evil do not go home to stroke a white cat and cackle at their own evilness. They think they are the good guys and that they do what's necessary to achieve their good goals. We think they're wrong, but that's an outside view. As has been pointed out, the road to hell is never in need of repair.

Given this, it's useful to have firebreaks, boundaries which serve to stop really determined people who think they're doing good from doing too much evil. A major firebreak is emotional empathy -- it serves as a check on runaway optimization processes which are, of course, subject to the Law of Unintended Consequences.

And, besides, I like humans more than I like optimization algorithms :-P

This is actually something I've been trying to articulate for a long time. It's fantastic to finally have a scientific name for it, (emotional vs cognitive empathy) along with a significantly different perspective.

I'd be inclined to share this outside the rationalist community. Ideally, me or someone else would weave most of the same concepts into a piece intellectuals in general as a target audience. (NOT someone associated directly with EA though, and not with too much direct discussion of EA, because we wouldn't want to taint it as a bunch of straw Vulcans.)

However, this is well written and might suffice for that purpose. The only things I think would confuse random people linked to this would be the little Hanson sitting on your shoulder, the EY empathy/saving the world bit, and the mention of artificial intelligence. It might also not be clear that your argument is quite narrow scope. (You're only criticizing some forms of emotional empathy, not all forms, and not cognitive empathy. You aren't, for instance, arguing against letting emotional empathy encourage us to do good in the first place, but only against letting it overpower the cognitive empathy that would let us do good effectively.)

So, does anyone have any thoughts as to whether linking non-nerds to this would still be a net positive? I guess the value of information is high here, so I can share with a few friends as an experiment. Worst case is I spend a few idiosyncrasy credits/weirdness points.

This post doesn't have much that addresses the "expanding circle" case for empathy, which goes something like this:

Empathy is a powerful tool for honing in on what matters in the world. By default, people tend to use it too narrowly. We can see that in many of the great moral failings of the past (like those mentioned here) which involved people failing to register some others as an appropriate target of empathy, or doing a lousy job of empathizing which involved making up stories more than really putting oneself in their shoes, or actively working to block empathy by dehumanizing them and evoking disgust, fear, or other emotions. But over time there has been moral progress as societies have expanded the circle of who people habitually feel empathy for, and developed norms and institutions to reflect their membership in that circle of concern. And it is possible to do better than your societal default if you cultivate your empathy, including the ability to notice the blind spots where you could be empathizing but are not (and the ability to then direct some empathy towards those spots). This could include people who are far away or across some boundary, people in an outgroup who you might feel antagonistic towards, people who have been accused of some misdeed, people and nonhumans that are very different from you, those who are not salient to you at the moment, those who don't exist yet, those who are only indirectly affected by your actions, etc.

I am afraid I cannot claim here any particularly noble motives.

In Jacobian's text there are, basically, two decision points: the first one is deciding to do good, and the second one is deciding on a course of action. You lose empathy in between them. There are (at least) two ways to interpret this. In one when you decide "do good", you make just a very generic decision to do some unspecified good. All the actual choices are at the "course of action" point. In another one at the first decision point you already decide what particular good do you want to work towards and then the second decision point is just the details of implementation.

I didn't want to start dissecting Jacobian's post at this level of detail, so I basically simplified it by saying that you lose your empathy before making some (but not necessarily all) choices. I don't know if you want to classify it as "technically incorrect" :-/

I think this is the first article in a long time that straight up changed my opinion in a significant way. I always considered empathy a universally good thing – in all forms. In fact I held it as one of the highest values. But the logic of the article is hard to argue with.

I still tentatively disagree that it [emotional empathy] inherently bad. Following what I read, I'd say it's harmful because it's overvalued/misunderstood. The solution would be to recognize that it's an egoistical thing – as I'm writing this I can confirm that I think this now. Whereas cognitive empathy is the selfless thing.

Doing more self-analysis, I think I already understood this on some level, but I was holding the concept of empathy in such high regards that I wasn't able to consciously criticize it.

I think this article is something that people outside of this community really ought to read.

So, if the emotional empathy should be discarded, why should I help all those strangers? The only answer that the link suggests is "social propriety".

But social propriety is a fickle thing. Sometimes it asks you to forgive the debts of the destitute, and sometimes it asks you to burn the witches. Without empathy, why shouldn't you cheer at the flames licking the evil witch's body? Without empathy, if there are some kulaks or Juden standing in the way of the perfect society, why shouldn't you kill them in the most efficient manner at your disposal?

I wrote a long post encouraging people to read The Sequences and getting slightly miffed at "postrationalists" on my blog.