dlthomas comments on Holden's Objection 1: Friendliness is dangerous - Less Wrong

11 Post author: PhilGoetz 18 May 2012 12:48AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (428)

You are viewing a single comment's thread. Show more comments above.

Comment author: dlthomas 18 May 2012 05:28:58PM 2 points [-]

My understanding is that CEV is based on consensus, in which case the majority is meaningless.

Comment author: steven0461 18 May 2012 08:39:01PM *  6 points [-]

Some quotes from the CEV document:

Coherence is not a simple question of a majority vote. Coherence will reflect the balance, concentration, and strength of individual volitions. A minor, muddled preference of 60% of humanity might be countered by a strong, unmuddled preference of 10% of humanity. The variables are quantitative, not qualitative.

(...)

It should be easier to counter coherence than to create coherence.

(...)

In qualitative terms, our unimaginably alien, powerful, and humane future selves should have a strong ability to say "Wait! Stop! You're going to predictably regret that!", but we should require much higher standards of predictability and coherence before we trust the extrapolation that says "Do this specific positive thing, even if you can't comprehend why."

Though it's not clear to me how the document would deal with Wei Dai's point in the sibling comment. In the absence of coherence on the question of whether to protect, persecute, or ignore impopular minority groups, does CEV default to protecting them or ignoring them? You might say that as written, it would obviously not protect them, because there was no coherence in favor of doing so; but what if protection of minority groups is a side effect of other measures CEV was taking anyway?

(For what it's worth, I suspect that extrapolation would in fact create enough coherence for this particular scenario not to be a problem.)

Comment author: dlthomas 18 May 2012 08:56:29PM 0 points [-]

Thank you. So, not quite consensus but similarly biased in favor if inaction.

Comment author: Wei_Dai 18 May 2012 06:40:56PM 4 points [-]

My understanding is that CEV is based on consensus, in which case the majority is meaningless.

If CEV doesn't positively value some minority group not being killed (i.e., if it's just indifferent due to not having a consensus), then the majority would be free to try to kill that group. So we really do need CEV to saying something about this, instead of nothing.

Comment author: dlthomas 18 May 2012 06:42:45PM 0 points [-]

Assuming we have no other checks on behavior, yes. I'm not sure, pending more reflection, whether that's a fair assumption or not...

Comment author: DanArmak 19 May 2012 04:00:28PM 2 points [-]

There is absolutely no reason to think that the values of all humans, extrapolated in some way, will arrive at a consensus.