eli_sennesh comments on Siren worlds and the perils of over-optimised search - Less Wrong

27 Post author: Stuart_Armstrong 07 April 2014 11:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (411)

You are viewing a single comment's thread. Show more comments above.

Comment author: [deleted] 30 April 2014 10:48:01AM *  0 points [-]

Assume B choose without coercion, but assume A always knows what B will choose and can set up various facts in the world to determine B's choice. Is B free?

So, just checking before I answer: you're claiming that no direct, gun-to-the-head coercion is employed, but Omega can always predict your actions and responses, and sets things up to ensure you will choose a specific thing.

Are you free, or are you in some sense "serving" Omega? I answer: The latter, very, very, very definitely.

If we take it out of abstract language, real people manipulate each-other all the time, and we always condemn it as a violation of the ethical principle of free choice. Yes, sometimes there are principles higher than free choice, as with a parent who might say, "Do your homework or you get no dessert" (treat that sentence as a metasyntactic variable for whatever you think is appropriate parenting), but we still prefer, all else equal, that our choices and those of others be less manipulated rather than more.

Just because fraud and direct coercion are the usual standards for proving a violation of free choice in a court of law, for instance in order to invalidate a legal contract, does not mean that these are the all-and-all of the underlying ethics of free choice.

Comment author: Stuart_Armstrong 30 April 2014 03:24:28PM 1 point [-]

Are you free, or are you in some sense "serving" Omega? I answer: The latter, very, very, very definitely.

Then if Omega is superintelligent, it has a problem: every single decision it makes increases or decreases the probability of someone answering something or other, possibly by a large amount. It seems Omega cannot avoid being coercive, just because it's so knowledgeable.

Comment author: [deleted] 01 May 2014 07:09:22AM 2 points [-]

We don't quite know that, and there's also the matter of whether Omega is deliberately optimizing those people or they're just reacting to Omega's optimizing the inanimate world (which I would judge to be acceptable and, yes, unavoidable).

Comment author: Stuart_Armstrong 02 May 2014 01:41:20PM 1 point [-]

It is an interesting question, though, and illustrates the challenges with "liberty" as a concept in these circumstances.

Comment author: [deleted] 02 May 2014 04:12:22PM 1 point [-]

Well yes. It's also why many people have argued in favor of Really Powerful Optimizers just... doing absolutely nothing.

Comment author: Stuart_Armstrong 06 May 2014 11:47:54AM 1 point [-]

That I don't see.