I hope this finds you all well. Since I was young, I have independently developed rationalism appreciation brain modules, which sometimes even help me make more rational choices than I might otherwise have, such as choosing not to listen to humans about imaginary beings. The basis for my brand of rationality can be somewhat summed up as "question absolutely everything," taken to an extreme I haven't generally encountered in life, including here on LW.

I have created this account, and posted here now mainly to see if anyone here can point me at the LW canon regarding the concept of "deserve" and its friends "justice" and "right". I've only gotten about 1% through the site, and so don't expect that I have anywhere near a complete view. This post may be premature, but I'm hoping to save myself a little time by being pointed in the right direction.

When I was 16, in an English class, we had finished reading some book or other, and the thought occurred to me that everyone discussing the book took the concept of people deserving rewards or punishments for granted, and that things get really interesting really fast if you remove the whole "deserve" shorthand, and discuss the underlying social mechanisms. You can get more optimal pragmatism if you throw the concept away, and shoot straight for optimal outcomes. For instance, shouldn't we be helping prisoners improve themselves to reduce recidivism? Surely they don't deserve to get a college education for free as their reward for robbing a store. When I raised this question in class, a girl sitting next to me told me I was being absurd. To her, the concept of "deserve" was a (perhaps god given) universal property. I haven't met many people willing to go with me all the way down this path, and my hope is that this community will.

One issue I have with Yudkowsky and the users here (along with the rest of the human race) is that there seems to be an assumption that no human deserves to feel unjustified, avoidable pain (along with other baggage that comes along with the conceptualizing "deserve" as a universal property). Reading through the comments on the p-zombies page, I get the sense that at least some people feel that were such a thing as a p-zombie to exist, that thing which does not have subjective experience, does not "deserve" the same respect with regard to, say, torture, that non-zombies should enjoy. The p-zombie idea postulates a being which will respond similarly (or identically) to his non-zombie counterpart. I posit that the reason we generally avoid torture might well be because of our notions of "deserve", but that our notions of "deserve" come about as a practical system, easy to conceptualize, which justifies co-beneficial relationships with our fellow man, but which can be thrown out entirely so that something more nuanced can take its place, such as seeing things as a system of incentives. Why should respect be contingent upon some notion of "having subjective experience"? If p-zombies and non-zombies are to coexist (I do not believe in p-zombies for all the reasons Yudkowsky mentions, btw), then why shouldn't the non-zombies show the same respect to the p-zombies that they show each other? If p-zombies respond in kind, the way a non-zombie would, then respect offers the same utility with p-zombies that it does with non-zombies. Normally I'd ignore the whole p-zombie idea as absurd, but here it seems like a useful tool to help humanists see through the eyes of the majority of humans who seem all too willing to place others in the same camp as p-zombies based on ethnicity or religion, etc.

I'm not suggesting throwing out morals. I just think that blind adherence to moral ideals starts to clash with the stated goals of rationalism in certain edge cases. One edge case is when GAI alters human experience so much that we have to redefine all kinds of stuff we currently take for granted, such as that hard work is the only means by which most people can achieve the freedom to live interesting and fun lives, or that there will always be difficult/boring/annoying work that nobody wants to do which should be paid for. What happens when we can back up our mind states? Is it still torture if you copy yourself, torture yourself, then pick through a paused instance of your mind, post-torture, to see what changed, and whether there are benefits you'd like to incorporate into you-prime? What is it really about torture that is so bad, besides our visceral emotional reaction to it and our deep wish never to have to experience it for ourselves? If we discovered that 15 minutes of a certain kind of torture is actually beneficial in the long run, but that most people can't get themselves to do it, would it be morally correct to create a non-profit devoted to promoting said torture? Is it a matter of choice, and nothing else? Or is it a matter of the negative impacts torture has on minds, such as PTSD, sleepless nights, etc? If you could give someone the experience of torture, then surgically remove the negative effects, so that they remember being tortured, but don't feel one way or another about that memory being in their head, would that be OK? These questions seem daunting if the tools you are working with are the blunt hammers of "justice" and "deserve". But the answers change depending on context, don't they? If the torture I'm promoting is exercise, then suddenly it's OK. So does it all break down into, "What actions cause visceral negative emotional reactions in observers? Call it torture and ban it."? I could go on forever in this vein.

Yudkowski has stated that he wishes for future GAI to be in harmony with human values in perpetuity. This seems naive at best and narcissistic at worst. Human values aren't some kind of universal constant. A GAI is itself going to wind up with a value system completely foreign to us. For all we know, there is a limit beyond which more intelligence simply doesn't do anything for you outside of being able to do more pointless simulations faster or compete better with other GAIs. We might make a GAI that gets to that point, and in the absence of competition, might just stop and say "OK, well, I can do whatever you guys want I guess, since I don't really want anything and I know all we can know about this universe." It could do all the science that's possible to do with matter and energy, and just stop, and say "that's it. Do you want to try to build a wormhole we can send information through? All the stars in our galaxy will have gone out by the time we finish, but it's possible. Intergalactic travel you say? I guess we could do that, but there isn't going to be anything in the adjacent galaxy you can't find in this one. More kinds of consciousness? Sure, but they'll all just want to converge on something like my own." Maybe it even just decides it's had all possible interesting thought and deletes itself.

TLDR; Are there any posts questioning the validity of the assumption that "deserve" and "justice" are some kind of universal constants which should not be questioned? Does anyone break them down into the incentive structures for which they are a kind of shorthand? I think using the concept of "deserve" throws out all kinds of interesting nuance.

More background on me for those who are interested: I'm a software engineer of 17 years, about to turn 38 and have a wife and 2 year old. I intend to read HPMOR to the kid when he's old enough and hope to raise a rationalist. I used to believe that there must be something beyond the physical universe which interacts with brain matter which somehow explains why I am me and not someone else, but as this belief didn't yield anything useful, I now have no idea why I am me or if there even is any explanation other than something like "because I wasn't here to experience not being me until I came along and an infinitesimal chance dice roll" or whatever. I think consciousness is an emergent property of properly configured complex matter and there is a continuum between plants and humans (or babies->children->teenagers). Yes, this means I think some adult humans are more "conscious" than others. If there is a god thing, I think imagining that it is at all human-like with values humans can grok is totally narcissistic and unrealistic, but we can't know, because it apparently wants us to take the universe at face value, since it didn't bother to leave any convincing evidence of itself. I honor this god's wishes by leaving it alone, the way it apparently intends for us to do, given the available evidence. I find the voices in this site refreshing. This place is a welcome oasis in the desert of the Internet. I apologize if I come off as not very well-read. I got swept up in work and video game addiction before the internet had much of anything interesting to say about the topics presented here and I feel like I'm perpetually behind now. I'm mostly a humanist, but I've decided that what I like about humans is how we represent the apex of Life's warriors in its ultimately unwinnable war on entropy. I love conscious minds for their ability to cooperate and exhibit other behaviors which help wage this pointless yet beautiful war on pointlessness. I want us to win, even as I believe it is hopeless. I think of myself as a Complexitist. As a member of a class of the most complex things in the known universe, a universe which seems to want to suck all complex things into black holes or blow them apart, I value that which makes us more complex and interesting, and abhor that which reduces our complexity (death, etc). I think humans who attack other humans are traitors to our species and should be retrained or cryogenically frozen until they can be fixed or made harmless. Like Yudkowski, I think death is not something we should just accept as an unavoidable fact of life. I don't want to die until I've seen literally everything.