You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Qiaochu_Yuan comments on DRAFT:Ethical Zombies - A Post On Reality-Fluid - Less Wrong Discussion

0 Post author: MugaSofer 09 January 2013 01:38PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (116)

Comments (116)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: Qiaochu_Yuan 11 January 2013 12:29:01AM * 2 points [-]

I think I'm using "anthropic" in a way consistent with the end of the first paragraph of Fundamentals of kicking anthropic butt (to refer to situations in which agents get duplicated and/or there is some uncertainty about what agent an agent is). If there's a more appropriate word then I'd appreciate knowing what it is.

My first objection is already contained in Vladimir_Nesov's comment: it seems like in general anthropic problems should be phrased entirely as decision problems and not as problems involving the assignment of odds. For example, Sleeping Beauty can be turned into two decision problems: one in which Sleeping Beauty is trying to maximize the expected number of times she is right about the coin flip, and one in which Sleeping Beauty is trying to maximize the probability that she is right about the coin flip. In the first case, Sleeping Beauty's optimal strategy is to guess tails, whereas in the second case it doesn't matter what she guesses. In a problem where there's no anthropic funniness, there's no difference between trying to maximize the expected number of times you're right and trying to maximize the probability that you're right, but with anthropic funniness there is.

My second objection is that I don't understand how an agent could be convinced of the truth of a sufficiently bizarre premise. (I have the same issue with Pascal's mugging, torture vs. dust specks, and Newcomb's problem.) In this particular case, I don't understand how I could be convinced that another agent really has the capacity to perfectly simulate me. This seems like exactly the kind of thing that agents would be incentivized to lie about in order to trick me.

Comment author: Wei_Dai 19 January 2013 12:56:50AM 3 points [-]

My second objection is that I don't understand how an agent could be convinced of the truth of a sufficiently bizarre premise. (I have the same issue with Pascal's mugging, torture vs. dust specks, and Newcomb's problem.) In this particular case, I don't understand how I could be convinced that another agent really has the capacity to perfectly simulate me. This seems like exactly the kind of thing that agents would be incentivized to lie about in order to trick me.

You may eventually obtain the capacity to perfectly simulate yourself, in which case you'll run into similar issues. I used Omega in a scenario a couple of years ago that's somewhat similar to the OP's, but really Omega is just a shortcut for establishing a "clean" scenario that's relatively free of distractions so we can concentrate on one specific problem at a time. There is a danger of using Omega to construct scenarios that have no real-world relevance, and that's something that we should keep in mind, but I think it's not the case in the examples you gave.

Comment author: ESRogs 11 January 2013 02:35:30AM 2 points [-]

How would you characterize your issue with Pascal's mugging? The dilemma is not supposed to require being convinced of the truth of the proposition, just assigning it a non-zero probability.

Comment author: Qiaochu_Yuan 11 January 2013 02:55:43AM 3 points [-]

Hmm. You're right. Upon reflection, I don't have a coherent rejection of Pascal's mugging yet.

Comment author: ESRogs 11 January 2013 04:13:22AM 1 point [-]

Gotcha. Your posts have seemed pretty thoughtful so far so I was surprised by / curious about that comment. :)

Comment author: MugaSofer comment score below threshold [+] (13 children)

Comment author: Qiaochu_Yuan 11 January 2013 08:18:00PM 1 point [-]

How then, do you assign the odds?

You don't.

You believe Omega because it's Omega, who always tells the truth and has access to godlike power.

How does any particular agent go about convincing me that it's Omega?

Comment author: MugaSofer 13 January 2013 10:28:40AM -2 points [-]

How then, do you assign the odds?

You don't.

So ... you don't know what the odds are, but you know how to act anyway? I notice that I am confused.

it's Omega, who always tells the truth and has access to godlike power.

How does any particular agent go about convincing me that it's Omega?

Assuming that there is only one godlike agent known or predicted in the environment, that Omega is a known feature of the environment, and that you have no reason to believe that e.g. you are hallucinating, then presumably all Omega needs to do is demonstrate his godlike powers - by predicting your every action ahead of time, say, or turning the sky green with purple spots.

Comment author: Qiaochu_Yuan 13 January 2013 12:22:35PM 0 points [-]

So ... you don't know what the odds are, but you know how to act anyway? I notice that I am confused.

In an anthropic situation, I don't think it makes sense to assign odds to statements like "I will see X" because the meaning of the term "I" becomes unclear. (For example, I don't think it makes sense for Sleeping Beauty to assign odds to statements like "I will see heads.") I can still assign odds to statements like "exactly fifteen copies of me will see X" by reasoning about what I currently expect my copies to see, given what I know about how I'll be copied, and using those odds I can still make decisions.

Assuming that there is only one godlike agent known or predicted in the environment, that Omega is a known feature of the environment, and that you have no reason to believe that e.g. you are hallucinating, then presumably all Omega needs to do is demonstrate his godlike powers - by predicting your every action ahead of time, say, or turning the sky green with purple spots.

Omega needs to both always tell the truth and have access to godlike power. How does Omega prove to me that it always tells the truth?

Comment author: MugaSofer comment score below threshold [+] (8 children)

Comment author: Qiaochu_Yuan 13 January 2013 05:19:25PM 0 points [-]

What do you think Alice should choose?

There's nothing to choose until you've specified a decision problem.

It is a known feature of the environment that people are regularly ambushed by an agent, calling itself Omega, which has never yet been known to lie and has access to godlike power.

An agent with godlike power can manufacture evidence that it has any other traits it wants, so observing such evidence isn't actually evidence that it has those traits.

Comment author: MugaSofer comment score below threshold [+] (6 children)

Comment author: Qiaochu_Yuan 13 January 2013 07:41:46PM * 0 points [-]

Oh, you were referring to the original decision problem. I was referring to the part of the post I was originally responding to about what subjective odds Alice should assign things. I just ran across an LW post making an important comment on these types of problems, which is that the answer depends on how altruistic Alice feels towards copies of herself. If she feels perfectly altruistic towards copies of herself, then sure, take the cakes.

An agent with godlike power can manufacture evidence of anything. This seems suspiciously like a Fully General Counterargument. Such an agent could, in any case, directly hack your brain so you don't realize it can falsify evidence.

Yes, it's a fully general counterargument against believing anything that an agent with godlike power says. Would this sound more reasonable if "godlike" were replaced with "devil-like"?

Comment author: MugaSofer 13 January 2013 09:18:35PM -2 points [-]

I just ran across an LW post making an important comment on these types of problems, which is that the answer depends on how altruistic Alice feels towards copies of herself. If she feels perfectly altruistic towards copies of herself, then sure, take the cakes.

That's ... a very good point, actually.

I assume, on this basis, that you agree with the hypothetical at the end?

Would this sound more reasonable if "godlike" were replaced with "devil-like"?

Yes. Devil-like implies known hostility.

By this logic, the moment you turn on a provably Friendly AI you should destroy it, because it might have hacked you into thinking it's friendly. Worse still, a hostile god would presumably realize you wont believe it, and so hack you into thinking it's not godlike; so anything claiming not to be godlike is lying.

Bottom line: gods are evidence that the world may be a lie. But not strong evidence.

continue this thread »

Comment author: Decius 12 January 2013 02:27:56AM -1 points [-]

How does any particular agent go about convincing me that it's Omega?

I don't know, but Omega does. Probably by demonstrating the ability to do something such that you believe the chance that it could be faked are epsilon^2, where epsilon is your prior belief that a given agent could have godlike powers.

Qiaochu_Yuan comments on DRAFT:Ethical Zombies - A Post On Reality-Fluid - Less Wrong Discussion

Article Navigation

Comments (116)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Open Threads

Recent Rationality Diaries

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (116)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login