wedrifid comments on Games for Rationalists - Less Wrong

5 Post author: Gunnar_Zarncke 12 September 2013 05:41PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (55)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 14 September 2013 01:42:58AM 1 point [-]

I wonder what this means in the context of FAI and what EY wrote on coherent extrapolated volition of humanity. That might be read to imply that the values of all humans count equally.

That's an interesting question and one where I am wary of EY's expressed beliefs on the subject. Eliezer has only ever talked about CEV as if it is something that applies to "humanity", without specifying the aggregation mechanism but with the implication that once Extrapolation is applied the humans would in general have acceptable values. But this opens up enormous scope for the "What if most people are Dicks?" problem as well as the perverse incentive that people who breed more now can expect CEV to be more in favour of them. I'm not entirely confident that CEV<humanity> wouldn't result in a genocide or two and at best there are going to be more mild dystopic outcomes.

Optimising for CEV<humanity> may sound egalitarian but on close inspection it is inherently less egalitarian than optimising for CEV<an actually egalitarian individual>. (This example should not be interpreted as an advocacy of creating an FAI with that value system.)

But social hierarchies and http://lesswrong.com/lw/hv6/gains_from_trade_slug_versus_galaxy_how_much/ among others implies that values of high status individuals may count more.

It could. But it needn't. Just like we are not obliged to maximise our inclusive genetic fitness just because that was the incentives of our genes as we were evolving we need not implement a CEV based on social status just because currently human influence follows that pattern. There are two parts to this consideration: it applies both to determining the aggregation mechanism used by the CEV procedure itself and also, for a given aggregation procedure and given target CEV may compute a system based somewhat on status. This again makes us look at which group CEV is applied to. CEV<a group of nerds> will likely emphasise status to a different degree than CEV<humanity> and so to the extent that those are different a group of nerds can be expected to be wary of CEV<humanity>.