All of Prolorn's Comments + Replies

Looks to me like Bostrom is trying to make the point that duplication of brain-states, by itself and devoid of other circumstances, is not sufficient to make the act of torture moral, or less harmful. After reading through the paper, it looks to me like we've moved outside of what Bostrom was trying to address, here. If synchronized brains lose individuality, and/or an integration process takes place, leading to a brain-state which has learned from the torture experience but remains unharmed, move the argument outside the realm of what Bostrom was trying to address. I agree with Bostrom on this point. It looks to me like, if Yorik is dismissing 49 tortured copies as inconsequential, he must also show that there is a process where the knowledge accumulated by each of the 49 copies is synchronized and integrated into the remaining one copy, without causing that one copy (or anyone else, for that matter) any harm. Or, there must be some other assumptions that he is making about the copies that remove the damage caused by copying - copying alone can't remove responsibility for the killing of the copies. For the black-hole example, copying the person about to be sucked into the hole is not ethically meaningless. The value of the copy, though, comes from its continued existence. The act of copying does not remove moral consequences from the sucking-in-the-black-hole act. If there is an agent X which pushed the copy into the black hole, that agent is just as responsible for his actions if he doesn't copy the individual at the last minute, as he would be if he does make a copy.

Can you please point me to Bostrom's paper? I can't seem to find the reference. I'm very curious if the in-context quote is better fleshed out. As it stands here, it looks a lot like it's affected by anthropomorphic bias (or maybe references a large number of hidden assumptions that I don't share, around both the meaning of individuality and the odds that intelligences which regularly undergo synchronization can remain similar to ours). I can imagine a whole space of real-life, many-integrated-synchronized-copies scenarios, where the process of creating a copy and torturing it for kicks would be accepted, commonplace and would not cause any sort of moral distress. To me, there is a point where torture and/or destruction of a synchronized, integrated, identical copy transition into the same moral category as body piercings and tatoos.

No. I grant special status to exceedingly unique minds, and to the last few of a given species. But human minds are very similar to each other, and granting different moral status to different humans is a very dangerous game. Here, I am looking at the practical effects of moral systems (Eliezer's post on "running on corrupted hardware" is relevant). The thoeretical gains of treating humans as having varrying moral status are small; the practical risks are huge (especially as our societies, though cash, reputation and other factors, is pretty good at distinguishing between people without having to further grant them different moral status). One cannot argue: "I agree with moral system M, but M has consequence S, and I disagree with S". Hence I cannot agree with granting people different moral status, once they are sufficiently divergent.