= 27d567bc2d48d9f40e9ccc20ea370b15)
nazgulnarsil comments on BOOK DRAFT: 'Ethics and Superintelligence' (part 1) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (107)
I have never understood what is wrong with the amnesia-holodecking scenario. (is there a proper name for this?)
If you want to, say, stop people from starving to death, would you be satisfied with being plopped on a holodeck with images of non-starving people? If so, then your stop-people-from-starving-to-death desire is not a desire to optimize reality into a smaller set of possible world-states, but simply a desire to have a set of sensations so that you believe starvation does not exist. The two are really different.
If you don't understand what I'm saying, the first two paragraphs of this comment might explain it better.
thanks for clarifying. I guess I'm evil. It's a good thing to know about oneself.
Uh, that was a joke, right?
no.
What definition of evil are you using? I'm having trouble understanding why (how?) you would declare yourself evil, especially evil_nazgulnarsil.
i don't care about suffering independent of my sensory perception of it causing me distress.
Oh. In that case, it might be more precise to say that your utility function does not assign positive or negative utility to the suffering of others (if I'm interpreting your statement correctly). However, I'm curious about whether this statement holds true for you at extremes, so here's a hypothetical.
I'm going to assume that you like ice cream. If you don't like any sort of ice cream, substitute in a certain quantity of your favorite cookie. If you could get a scoop of ice cream (or a cookie) for free at the cost of a million babies thumbs cut off, would you take the ice cream/cookie?
If not, then you assign a non-zero utility to others suffering, so it might be true that you care very little, but it's not true that you don't care at all.
I think you misunderstand slightly. Sensory experience includes having the idea communicated to me that my action is causing suffering. I assign negative utility to other's suffering in real life because the thought of such suffering is unpleasant.
Alright. Would you take the offer if Omega promised that he would remove your memories of the agreement of having a million babies' thumbs cut off for a scoop of ice cream right after you made the agreement, so you could enjoy your ice-cream without guilt?
Well, it's essentially equivalent to wireheading.
which I also plan to do if everything goes tits-up.