You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Qiaochu_Yuan comments on DRAFT:Ethical Zombies - A Post On Reality-Fluid - Less Wrong Discussion

0 Post author: MugaSofer 09 January 2013 01:38PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (116)

Comments (116)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: Qiaochu_Yuan 13 January 2013 05:19:25PM 0 points [-]

What do you think Alice should choose?

There's nothing to choose until you've specified a decision problem.

It is a known feature of the environment that people are regularly ambushed by an agent, calling itself Omega, which has never yet been known to lie and has access to godlike power.

An agent with godlike power can manufacture evidence that it has any other traits it wants, so observing such evidence isn't actually evidence that it has those traits.

Comment author: MugaSofer comment score below threshold [+] (6 children)

Comment author: Qiaochu_Yuan 13 January 2013 07:41:46PM * 0 points [-]

Oh, you were referring to the original decision problem. I was referring to the part of the post I was originally responding to about what subjective odds Alice should assign things. I just ran across an LW post making an important comment on these types of problems, which is that the answer depends on how altruistic Alice feels towards copies of herself. If she feels perfectly altruistic towards copies of herself, then sure, take the cakes.

An agent with godlike power can manufacture evidence of anything. This seems suspiciously like a Fully General Counterargument. Such an agent could, in any case, directly hack your brain so you don't realize it can falsify evidence.

Yes, it's a fully general counterargument against believing anything that an agent with godlike power says. Would this sound more reasonable if "godlike" were replaced with "devil-like"?

Comment author: MugaSofer 13 January 2013 09:18:35PM -2 points [-]

I just ran across an LW post making an important comment on these types of problems, which is that the answer depends on how altruistic Alice feels towards copies of herself. If she feels perfectly altruistic towards copies of herself, then sure, take the cakes.

That's ... a very good point, actually.

I assume, on this basis, that you agree with the hypothetical at the end?

Would this sound more reasonable if "godlike" were replaced with "devil-like"?

Yes. Devil-like implies known hostility.

By this logic, the moment you turn on a provably Friendly AI you should destroy it, because it might have hacked you into thinking it's friendly. Worse still, a hostile god would presumably realize you wont believe it, and so hack you into thinking it's not godlike; so anything claiming not to be godlike is lying.

Bottom line: gods are evidence that the world may be a lie. But not strong evidence.

Comment author: Qiaochu_Yuan 13 January 2013 11:38:55PM 0 points [-]

I assume, on this basis, that you agree with the hypothetical at the end?

Nope. I still don't agree that simulating an agent N times and doing X to one of them is morally equivalent to risking X to them with probability 1/(N+1). For example, if you are not at all altruistic to copies of yourself, then you don't care about the former situation as long as the copy that X is being done to is not you. On the other hand, if you value fairness among your copies (that is, if you value your copies having similar quality of life) then you care about the former situation more strongly than the latter situation.

By this logic, the moment you turn on a provably Friendly AI you should destroy it, because it might have hacked you into thinking it's friendly. Worse still, a hostile god would presumably realize you wont believe it, and so hack you into thinking it's not godlike; so anything claiming not to be godlike is lying.

Pretty much the only thing a godlike agent can convince me of is that it's godlike (and I am not even totally convinced this is possible). After that, again, whatever evidence a godlike agent presents of anything else could have been fabricated. Your last inference doesn't follow from the others; my priors regarding the prevalence of godlike agents is currently extremely low, and claiming not to be godlike is not strong evidence either way.

Comment author: MugaSofer 14 January 2013 10:04:22AM * -2 points [-]

Nope. [snip explanation]

To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn't it be equivalent to Alice-who-values-copies-of-herself?

my priors regarding the prevalence of godlike agents is currently extremely low,

And what are those priors based on? Evidence! Evidence that a godlike being would be motivated to falsify!

Comment author: Qiaochu_Yuan 14 January 2013 09:32:04PM 0 points [-]

To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn't it be equivalent to Alice-who-values-copies-of-herself?

Sure, but the result you describe is equivalent to Alice being an average utilitarian with respect to copies of herself. What if Alice is a total utilitarian with respect to copies of herself?

Comment author: MugaSofer 15 January 2013 10:27:28AM -2 points [-]

Actually, she should still make the same choice, although she would choose differently in other scenarios.

Qiaochu_Yuan comments on DRAFT:Ethical Zombies - A Post On Reality-Fluid - Less Wrong Discussion

Article Navigation

Comments (116)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Open Threads

Recent Rationality Diaries

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (116)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login