The author is far from alone in his view that both a complete rightness criterion and a consistent decision method must be required of all serious moral theories.

Among hedonistic utilitarians it's quite normal to demand both completeness, to include all (human) situations, and consistency, to avoid contradictions. The author simply describes what's normal among consequentialists, who, after all, are more or less the rational ones. ;-) There's one interesting exception though! The demand to include all situations, including the non-human ones, is radical, and quite hard a challenge for hedonistic utilitarians, who do have problems with the bloodthirsty predators of the jungle.

I don't think I would need to define it that way for the above comment to be coherent.

The author says a moral theory should:

• "Cover how one should act in all situations" (instead of dealing only with 'moral' ones)
• Contain no contradictions
• "Cover all situations in which somebody should perform an action, even if this “somebody” isn’t a human being"

In other words, a decision theory, complete with an algorithm (so you can actually use it), and a full set of terminal goals. Not what anyone else means by "moral theory'.

The success of a Facebook group depends a lot on how it get's promoted and whether there are a few people who care about creating content for it.

If it's a tool AGI, I don't see how it would help with friendliness, and if it's an active self-developing AGI, I thought the canonical position of LW was that there could be only one? and it's too late to do anything about friendliness at this point?

Options (b) and (c) are basically wishes and those are complex X-D

"Not kill us" is an easy criterion, we already have an AI like that, it plays Go well.

Nothing, because we still don't know what a friendly AI is.

Nothing. I don't think facebook membership counts are a good measurement.

I agree with you on the complexity of value. However, perhaps we are imagining the ideal way of aggregating all those complex values differently. I absolutely agree that the simple models I keep proposing for individual values are spherical cows, and ignore a lot of nuance. I just don't see things working radically differently when the nuance is added in, and the values aggregated.

That sounds like a really complex discussion though, and I don't think either of us is likely to convince the other without a novel's worth of text. However, perhaps I can convince you that you already are suppressing some impulses, and that this isn't always disastrous. (Though it certainly can be, if you choose the wrong ones.)

there aren't large benefits to be gained by discarding some emotions and values.

Isn't that what akrasia is?

If I find that part of me values one marshmallow now at the expense of 2 later, and I don't endorse this upon reflection, wouldn't it make sense to try and decrease such impulses? Removing them may be unnecessarily extreme, but perhaps that's what some nootropics do.

Similarly, if I were to find that I gained a sadistic pleasure from something, I wouldn't endorse that outside of well defined S&M. If I had an alcoholism problem, I'd similarly dislike my desire for alcohol. I suspect that strongly associating cigarettes with disgust is helpful in counteracting the impulse to smoke.

If I understand correctly, some Buddhist try to eliminate suffering by eliminating their desires. I find this existentially terrifying. However, I think that boosting and suppressing these sorts of impulses is precisely what psychologists call conditioning. A world where none refines or updates their natural impulses is just as unsettling as the Buddhist suppression of all values.

So, even if you don't agree that there are cases where we should suppress certain pro-social emotions, do you agree with my characterization of antisocial emotions and grey area impulses like akrasia?

(I'm using values, impulses, emotions, etc fairly interchangeably here. If what I'm saying isn't clear, let me know and I can try to dig into the distinctions.)

These six principles are true as far as they go, but I feel they're so weak so not to be very useful. I'd like to offer a more cynical view.

The article's goal is, more or less, to avoid being convinced of untrue things by motivated agents. This has a name: Defense Against the Dark Arts. And I feel like these six principles are about as effective in real life as taking the canonical DADA first year class and then going up against HPMOR Voldemort.

With today's information technology and globalization, we're all exposed to world-class Dark Arts practitioners. Not being vulnerable to Cialdini's principles might help defend you in an argument with your coworker. But it won't serve you well when doubting something you read in the news or in an FDA-endorsed study.

And whatever your coworker or your favorite blog was arguing probably derives from such a curated source to begin with. All arguments rest on factual beliefs - outside of math anyway - and most of us are very far from being able to verify the facts we believe. And your own prior beliefs need to be well supported, to avoid being rejected on the same basis.