Interesting thread!
I'm not sure that pjeby has fully adressed Eliezer's concern that "eliminating my negative emotions would be changing my preferences, and changing my preferences so that they're satisfied is against my current preferences (otherwise, I'd just go for being an orgasmium)".
(Well, at least that's how I'd paraphrase it, Eliezer, tell me if I'm wrong)
To which I would answer:
Yes, it's very possible that eliminating some negative emotions would be immoral, or at least, would change one's preferences in a way my previous preferences would disagree with (think: eliminating the guilt over killing people, and things like that. I wouldn't be very happy to learn that the army or police of a dictatorship is researching emotion elimination)
Still, there is probably a wide range of negative feelings that could be removed in a way that doesn't contradict one's original preferences - in the sense that the pre-modification person wouldn't find the behaviour of the modified person objectionable.
The line between which changes are OK and which are not is not that obvious to draw, and many posts on OB talk about it (The difference between the morality of the ancient greek and our own, and thus the risk of "freezing" our own morality and barring future moral progress, the Confessor's objections to non-consensual sex, etc.). pjeby might be being a bit light-handed when he dismisses concerns over changing preferences as "irrational", but I think he meant that careful examination could show that those changes stayed in the second category and wouldn't turn one into a immoral monster.
(It feels a bit weird answering pjeby's post in the third person, but it felt clearer to me that way :P I'm not responding to this post in particular)
(Disclaimer: I'm one of pjeby's clients, but that's not why I'm here, I've been reading OvercomingBias since nearly the beginning)
pjeby might be being a bit light-handed when he dismisses concerns over changing preferences as "irrational"
I didn't (explicitly) dismiss those concerns; I said that away-from reasoning has a higher rationality standard to meet, in part because it's likely to be vague.
I wasn't even thinking about preference-changing being dangerous, because our preferences are largely independent and mostly don't "auto-update" when we change one -- there's a LOT of redundancy. So if a specific change isn't compatible with your overall morality, you'...
To paraphrase the Black Belt Bayesian: Behind every exciting, dramatic failure, there is a more important story about a larger and less dramatic failure that made the first failure possible.
If every trace of religion was magically eliminated from the world tomorrow, then—however much improved the lives of many people would be—we would not even have come close to solving the larger failures of sanity that made religion possible in the first place.
We have good cause to spend some of our efforts on trying to eliminate religion directly, because it is a direct problem. But religion also serves the function of an asphyxiated canary in a coal mine—religion is a sign, a symptom, of larger problems that don't go away just because someone loses their religion.
Consider this thought experiment—what could you teach people that is not directly about religion, which is true and useful as a general method of rationality, which would cause them to lose their religions? In fact—imagine that we're going to go and survey all your students five years later, and see how many of them have lost their religions compared to a control group; if you make the slightest move at fighting religion directly, you will invalidate the experiment. You may not make a single mention of religion or any religious belief in your classroom, you may not even hint at it in any obvious way. All your examples must center about real-world cases that have nothing to do with religion.
If you can't fight religion directly, what do you teach that raises the general waterline of sanity to the point that religion goes underwater?
Here are some such topics I've already covered—not avoiding all mention of religion, but it could be done:
But to look at it another way—
Suppose we have a scientist who's still religious, either full-blown scriptural-religion, or in the sense of tossing around vague casual endorsements of "spirituality".
We now know this person is not applying any technical, explicit understanding of...
When you consider it—these are all rather basic matters of study, as such things go. A quick introduction to all of them (well, except naturalistic metaethics) would be... a four-credit undergraduate course with no prerequisites?
But there are Nobel laureates who haven't taken that course! Richard Smalley if you're looking for a cheap shot, or Robert Aumann if you're looking for a scary shot.
And they can't be isolated exceptions. If all of their professional compatriots had taken that course, then Smalley or Aumann would either have been corrected (as their colleagues kindly took them aside and explained the bare fundamentals) or else regarded with too much pity and concern to win a Nobel Prize. Could you—realistically speaking, regardless of fairness—win a Nobel while advocating the existence of Santa Claus?
That's what the dead canary, religion, is telling us: that the general sanity waterline is currently really ridiculously low. Even in the highest halls of science.
If we throw out that dead and rotting canary, then our mine may stink a bit less, but the sanity waterline may not rise much higher.
This is not to criticize the neo-atheist movement. The harm done by religion is clear and present danger, or rather, current and ongoing disaster. Fighting religion's directly harmful effects takes precedence over its use as a canary or experimental indicator. But even if Dawkins, and Dennett, and Harris, and Hitchens should somehow win utterly and absolutely to the last corner of the human sphere, the real work of rationalists will be only just beginning.