Open Thread June 2010, Part 3

Kevin

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

The thrilling conclusion of what is likely to be an inaccurately named trilogy of June Open Threads.

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

The thrilling conclusion of what is likely to be an inaccurately named trilogy of June Open Threads.

Dark arts, huh? Sometime ago I put forward the following scenario:

Bob wants to kill a kitten. The FAI wants to save the kitten because it's a good thing according to our CEV. So the FAI threatens Bob with 50 years of torture unless Bob lets the kitten go. The FAI has two distinct reasons why threatening Bob is okay: a) Bob will comply and there will be no need to torture him, b) the FAI is lying anyway. Expected utility reasoning says the FAI is doing the Right Thing. But do we want that?

(Yes, this is yet another riff on consequentialism, deontologism and lying. Should FAIs follow deontological rules? For that matter, should humans?)

Is that actually the FAI's only or best technique?

Off the top of my non-amplified brain:

Reward Fred for not torturing kittens.

Give Fred simulated kittens to torture and deny Fred access to real kittens.

Give Fred something harmless to do which he likes better than torturing kittens.

ETA Convince Fred that torturing kittens is wrong.

0Dagon16y

our CEV is (and has to be) detailed enough to answer the question of "do we want that?". Saving a kitten is a good thing. Being truthful to Bob is a good thing. Not torturing Bob is a good thing. The relative weights of these good things determines the FAI's actions. I'd say that the FAI should calculate some game-theoretic chance of torturing Bob for 50 years based on relative pain of kitten death and of having to inflict the torture. Depending on Bob's expected rationality level, we could tell him "you'll be tortured", or "you might be tortured", or the actual mechanism of determining whether he is tortured. Actually, strike that. Any competent AI will find ways aside from possible torture to make Bob not want that. Either agree with Bob's reason for killing the kitten, or fix him so he only wants things that make sense. I'm not sure how friendly this is - I haven't seen a good writeup or come to any conclusions myself of what FAI does with internal contradictions in a CEV (that is, when a population's extrapolated volition is not coherent).

5Nick_Tarleton16y

Expected utility reasoning with a particular utility function says the FAI is right. If we disagree, our preferences might be described by some other utility function.

9

Open Thread June 2010, Part 3

9

9

9

Open Thread June 2010, Part 3

9

9