Yes, the comment system here is really not suited to the kind of conversation I've been trying to have... not that I'm sure what system would work for it. ;-)
As far as meta-ethics goes, the short summary is:
"Avoiding badness" and "seeking goodness" are not interchangeable when you experience them concretely on human hardware,
It is therefore a reasoning error to treat them as if they were interchangeable in your abstract moral calculations (as they will not work the same way in practice),
Due to the specific nature of the human hardware biases involved (i.e., the respective emotional, chemical, and neurological responses to pain vs. pleasure) , badness-avoidance values are highly likely to be found irrational upon detailed examination... and thus they are always the ones worth examining first.
Badness-avoidance values are a disproportionately high (if not exclusive!) source of "motivated reasoning". i.e., we don't so much rationalize to paint pretty pictures, as to hide the ugly ones. (Which makes rooting them out of critical importance to rationalists.)
This summary is more to clarify my thoughts for the eventual post, than an attempt to continue the discussion. (To me, these things are so obvious and so much a part of my day-to-day experience that I often forget the inferential distance involved for most people.)
These ideas are all capable of experimental verification; the first one has certainly been written about in the literature. None are particularly unorthodox or controversial in and of themselves, as far as I'm aware.
However, there are common arguments against some of these ideas that my own students bring up, so in my (eventual) post I'll need to bring them up and refute them as well.
For example, a common argument against positively-motivated goodness is that feeling good about being generous means you're "really" being selfish... and thus bad! So, the person advancing this argument is motivated to rationalize the "virtue" of being dutiful -- i.e., doing something you don't want to, but nonetheless "should" -- because it would be bad not to.
Strangely, most people have these judgments only in relation to their self... They see no problem with someone else doing good out of generosity or kindness, with no pain or duty involved. It's only themselves they sentence to this "virtue" of suffering to achieve goodness. (Which is sort of like "fighting for peace" or "f*ing for virginity", but I digress.)
Whether this is something inbuilt, cultural, or selection bias of people I work with, I have no idea. But it's damn common... and Eliezer's making a virtue out of unhappiness (beyond the bare minimums demanded by safety, etc.) fits smack dab in the middle of this territory.
Whew. Okay, I'm going to stop writing this now... this really needs to be a post. Or several. The more I think about how to get here, starting from only the OB corpus and without recapitulating my own, the bigger I realize the inferential gap is.
This is what I was talking about. Please do prepare the posts, it'll help you to clarify your position to yourself. Let them lie as drafts for a while, then make a decision about whether to post them. Note that your statements are about the form of human preference computation, not about the utility that computes the "should" following from human preferences. Do you know the derivation of expected utility formula? You refer to a well-known finding that people avoid negative reward more than they seek positive reward.
To paraphrase the Black Belt Bayesian: Behind every exciting, dramatic failure, there is a more important story about a larger and less dramatic failure that made the first failure possible.
If every trace of religion was magically eliminated from the world tomorrow, then—however much improved the lives of many people would be—we would not even have come close to solving the larger failures of sanity that made religion possible in the first place.
We have good cause to spend some of our efforts on trying to eliminate religion directly, because it is a direct problem. But religion also serves the function of an asphyxiated canary in a coal mine—religion is a sign, a symptom, of larger problems that don't go away just because someone loses their religion.
Consider this thought experiment—what could you teach people that is not directly about religion, which is true and useful as a general method of rationality, which would cause them to lose their religions? In fact—imagine that we're going to go and survey all your students five years later, and see how many of them have lost their religions compared to a control group; if you make the slightest move at fighting religion directly, you will invalidate the experiment. You may not make a single mention of religion or any religious belief in your classroom, you may not even hint at it in any obvious way. All your examples must center about real-world cases that have nothing to do with religion.
If you can't fight religion directly, what do you teach that raises the general waterline of sanity to the point that religion goes underwater?
Here are some such topics I've already covered—not avoiding all mention of religion, but it could be done:
But to look at it another way—
Suppose we have a scientist who's still religious, either full-blown scriptural-religion, or in the sense of tossing around vague casual endorsements of "spirituality".
We now know this person is not applying any technical, explicit understanding of...
When you consider it—these are all rather basic matters of study, as such things go. A quick introduction to all of them (well, except naturalistic metaethics) would be... a four-credit undergraduate course with no prerequisites?
But there are Nobel laureates who haven't taken that course! Richard Smalley if you're looking for a cheap shot, or Robert Aumann if you're looking for a scary shot.
And they can't be isolated exceptions. If all of their professional compatriots had taken that course, then Smalley or Aumann would either have been corrected (as their colleagues kindly took them aside and explained the bare fundamentals) or else regarded with too much pity and concern to win a Nobel Prize. Could you—realistically speaking, regardless of fairness—win a Nobel while advocating the existence of Santa Claus?
That's what the dead canary, religion, is telling us: that the general sanity waterline is currently really ridiculously low. Even in the highest halls of science.
If we throw out that dead and rotting canary, then our mine may stink a bit less, but the sanity waterline may not rise much higher.
This is not to criticize the neo-atheist movement. The harm done by religion is clear and present danger, or rather, current and ongoing disaster. Fighting religion's directly harmful effects takes precedence over its use as a canary or experimental indicator. But even if Dawkins, and Dennett, and Harris, and Hitchens should somehow win utterly and absolutely to the last corner of the human sphere, the real work of rationalists will be only just beginning.