I don't think I would need to define it that way for the above comment to be coherent.

The author says a moral theory should:

• "Cover how one should act in all situations" (instead of dealing only with 'moral' ones)
• Contain no contradictions
• "Cover all situations in which somebody should perform an action, even if this “somebody” isn’t a human being"

In other words, a decision theory, complete with an algorithm (so you can actually use it), and a full set of terminal goals. Not what anyone else means by "moral theory'.

The success of a Facebook group depends a lot on how it get's promoted and whether there are a few people who care about creating content for it.

If it's a tool AGI, I don't see how it would help with friendliness, and if it's an active self-developing AGI, I thought the canonical position of LW was that there could be only one? and it's too late to do anything about friendliness at this point?

Options (b) and (c) are basically wishes and those are complex X-D

"Not kill us" is an easy criterion, we already have an AI like that, it plays Go well.

Nothing, because we still don't know what a friendly AI is.

Nothing. I don't think facebook membership counts are a good measurement.

I agree with you on the complexity of value. However, perhaps we are imagining the ideal way of aggregating all those complex values differently. I absolutely agree that the simple models I keep proposing for individual values are spherical cows, and ignore a lot of nuance. I just don't see things working radically differently when the nuance is added in, and the values aggregated.

That sounds like a really complex discussion though, and I don't think either of us is likely to convince the other without a novel's worth of text. However, perhaps I can convince you that you already are suppressing some impulses, and that this isn't always disastrous. (Though it certainly can be, if you choose the wrong ones.)

there aren't large benefits to be gained by discarding some emotions and values.

Isn't that what akrasia is?

If I find that part of me values one marshmallow now at the expense of 2 later, and I don't endorse this upon reflection, wouldn't it make sense to try and decrease such impulses? Removing them may be unnecessarily extreme, but perhaps that's what some nootropics do.

Similarly, if I were to find that I gained a sadistic pleasure from something, I wouldn't endorse that outside of well defined S&M. If I had an alcoholism problem, I'd similarly dislike my desire for alcohol. I suspect that strongly associating cigarettes with disgust is helpful in counteracting the impulse to smoke.

If I understand correctly, some Buddhist try to eliminate suffering by eliminating their desires. I find this existentially terrifying. However, I think that boosting and suppressing these sorts of impulses is precisely what psychologists call conditioning. A world where none refines or updates their natural impulses is just as unsettling as the Buddhist suppression of all values.

So, even if you don't agree that there are cases where we should suppress certain pro-social emotions, do you agree with my characterization of antisocial emotions and grey area impulses like akrasia?

(I'm using values, impulses, emotions, etc fairly interchangeably here. If what I'm saying isn't clear, let me know and I can try to dig into the distinctions.)

These six principles are true as far as they go, but I feel they're so weak so not to be very useful. I'd like to offer a more cynical view.

The article's goal is, more or less, to avoid being convinced of untrue things by motivated agents. This has a name: Defense Against the Dark Arts. And I feel like these six principles are about as effective in real life as taking the canonical DADA first year class and then going up against HPMOR Voldemort.

With today's information technology and globalization, we're all exposed to world-class Dark Arts practitioners. Not being vulnerable to Cialdini's principles might help defend you in an argument with your coworker. But it won't serve you well when doubting something you read in the news or in an FDA-endorsed study.

And whatever your coworker or your favorite blog was arguing probably derives from such a curated source to begin with. All arguments rest on factual beliefs - outside of math anyway - and most of us are very far from being able to verify the facts we believe. And your own prior beliefs need to be well supported, to avoid being rejected on the same basis.

Thanks for letting me know that CEV is obsolete. I'll have to look into the details. However, I don't think our disagreement is in that area.

it's easy to say we should increase happiness, all else being equal. It's not so obvious that we should increase it at the expense of other things

Agreed, but the argument works just as well for decreasing happiness as for possible increases. Even someone who valued their own happiness 1000x more than that of others would still prefer to suffer than for 1001 people to suffer. If they also value their own life 1000x as much as other people's lives, they would be willing to die to prevent 1001+ deaths. If you added up the total number of utils of happiness, according to his or her utility function, 99.9999% of the happiness they value would be happiness in other people, assuming there are on the order of billions of people and that they bite the bullet on the repugnant conclusion. (For simplicity's sake.)

But all that's really just to argue that there are things worth dying for, in the case of many people. My central argument looks something like this:

There are things worth dying for. Loosing something valuable, like by suppressing a biased emotion, is less bad than dying. If suppressing emotional empathy boosts the impact of cognitive empathy (I'm not sure it does) enough to achieve something worth dying for, then one should do so.

But I'm not sure things are so dire. The argument gets more charitable when re-framed as boosting cognitive empathy instead. In reality, I think what's actually going on is empathy either triggers something like near-mode thinking or far-mode, and these two possibilities are what leads to "emotional empathy" and "cognitive empathy". If so, then "discarding [emotional] empathy" seems far less worrying. It's just a cognitive habit. In principle though, if sacrificing something more actually was necessary for the greater good, then that would outweigh personal loss.