Caspian comments on [LINK] Scott Aaronson on Google, Breaking Circularity and Eigenmorality - Less Wrong

18 Post author: shminux 19 June 2014 08:17PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (46)

You are viewing a single comment's thread. Show more comments above.

Comment author: XiXiDu 20 June 2014 09:33:25AM 10 points [-]

Scott Says:

There’s a crucial observation that I took for granted in the post but shouldn’t have, so let me now make it explicit. The observation is this:

No system for aggregating preferences whatsoever—neither direct democracy, nor representative democracy, nor eigendemocracy, nor anything else—can possibly deal with the “Nazi Germany problem,” wherein basically an entire society’s value system becomes inverted to the point where evil is good and good evil.

Comment author: Caspian 28 June 2014 02:09:00AM 0 points [-]

If we're aggregating cooperation rather than aggregating values, we certainly can create a system that distinguishes between societies that apply an extreme level of noncooperation (i.e. killing) to larger groups of people than other societies, and that uses our own definition of noncooperation rather than what the Nazi values judge as noncooperation.

That's not to say you couldn't still find tricky example societies where the system evaluation isn't doing what we want, I just mean to encourage further improvement to cover moral behaviour towards and from hated minorities, and in actual Nazi Germany.