Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Kaj_Sotala 07 January 2018 04:17:53PM *  1 point [-]

I agree that the argument you advance here is the sane one, but I have trouble reconciling it with my interpretation of Effective Altruism: 'effort should be made to expend resources on preventing suffering, maximize the ratio of suffering avoided to cost expended'

I interpret your paper as rejecting the argument advanced by prof Hansen that if of all future variants of you, the number enjoying 'heaven' vastly outnumber the copies suffering 'hell', on balance, uploading is a good. Based on your paper's citation of Omelas, I assert that you would weight 'all future heaven copies' in aggregate, and all future hell copies individually.

Well, our paper doesn't really endorse any particular moral theory: we just mention a number of them, without saying anything about which one is true. As we note, if one is e.g. something like a classical utilitarian, then one would take the view by Hanson that you mention. The only way to really "refute" this is to say that you don't agree with that view, but that's an opinion-based view rather than a refutation.

Similarly, some people accept the various suffering-focused intuitions that we mention, while others reject them. For example, Toby Ord rejects the Omelas argument, and gives a pretty strong argument for why, in this essay (under the part about "Lexical Threshold NU", which is his term for it). Personally I find the Omelas argument very intuitively compelling, but at the same time I have to admit that Ord also makes a compelling argument against it.

That said, it's still possible and reasonable to end up accepting the Omelas argument anyway; as I said, I find it very compelling myself.

(As an aside, I tend to think that personal identity is not ontologically basic, so I don't think that it matters whose copy ends up getting tortured; but that doesn't really help with your dilemma.)

If you do end up with that result, my advice would be for you to think a few steps forward from the brain-shedding argument. Suppose that your argument is correct, and that nothing could justify some minds being subjected to torture. Does that imply that you should go around killing people? (The blender thing seems unnecessary; just plain ordinary death already destroys brains quite quickly.)

I really don't think so. First, I'm pretty sure that your instincts tell you that killing people who don't want to be killed, when that doesn't save any other lives, is something you really don't want to do. That's something that's at least worth treating as a strong ethical injunction, to only be overridden if there's a really really really compelling reason to do so.

And second, even if you didn't care about ethical injunctions, it looks pretty clear that going around killing people wouldn't actually serve your goal much - you'd just get thrown in prison pretty quickly, and also cause enormous backlash against the whole movement of suffering-focused ethics; anyone even talking about Omelas arguments would from that moment on get branded as "one of those crazy murderers" and everyone would try to distance themselves from them; which might just increase the risk of lots of people suffering from torture-like conditions, since a movement that was trying to prevent them would get discredited.

Instead, if you take this argument seriously, then what you should instead be doing is to try to minimize s-risks in general: if any given person ending up tortured would be one of the worst things that could happen, then large numbers of people ending up tortured would be even worse. We listed a number of promising-seeming approaches for preventing s-risks in our paper: none of them involve blenders, and several of them - like supporting AI alignment research - are already perfectly reputable within EA circles. :)

You may also want to read Gains from Trade through Compromise, for reasons to try to compromise and find mutually-acceptable solutions with people who don't buy the Omelas argument.

(Also, I have an older paper which suggests that a borg-like outcome may be relatively plausible, given that it looks like linking brains together into a borg could be relatively straightforward once we did have uploading - or maybe even before, if an exocortex prosthesis that could be used for mind-melding was also the primary uploading method.)

Comment author: RedMan 08 January 2018 02:10:42AM *  0 points [-]

An ethical injunction doesn't work for me in this context, killing can be justified with lots of more base motives than 'preventing infinity suffering'.

So, instead of a blender, I could sell hats with tiny brain pulping shaped charges that will be remotely detonated when mind uploading is proven to be possible, or when the wearer dies of some other cause. As long as my marketing reaches some percentage of people who might plausibly be interested, then I've done my part.

I assess that the number is small, and that anyone seriously interested in such a device likely reads lesswrong, and may be capable of making some arrangement for brain destruction themselves. So, by making this post and encouraging a potential upload to pulp themselves prior to upload. I have some > 0 probability of preventing infinity suffering.

I'm pretty effectively altruistic, dang. It's not even February.

I prefer your borg scenarios to individualized uploading. I feel like it's technically feasible using extant technology, but I'm not sure how much interest there really is in mechanical telepathy.

Comment author: Kaj_Sotala 05 January 2018 06:55:15AM 1 point [-]

Awesome paper.

Thank you very much!

Curious about your take on my question here: http://lesswrong.com/lw/os7/unethical_human_behavior_incentivised_by/

So, I agree that mind uploads being tortured indefinitely is a very scary possibility. And it seems very plausible that some of that is going to happen in a world with mind uploads, especially since it's going to be impossible to detect from the outside, unless you are going to check all the computations that anyone is running.

On the other hand, we don't know for sure what that world is going to be like. Maybe there will be some kind of AI in charge that does check everyone's computations, maybe all the hardware that gets sold is equipped with built-in suffering-detectors that disallow people from running torture simulations, or something. I'll admit that both of these seem somewhat unlikely or even far-fetched, but then again, someone might come up with a really clever solution that I just haven't thought of.

Your argument also seemed to me to have some flaws:

Over a long enough timeline, the probability of a copy of any given uploaded mind falling into the power of a sadistic jerk approaches unity. Once an uploaded mind has fallen under the power of a sadistic jerk, there is no guarantee that it will ever be 'free',

You can certainly make the argument that, for any event with non-zero probability, then over a sufficiently long lifetime that event will happen at some point. But if you are using that to argue that an upload will be captured by someone sadistic eventually, shouldn't you also hold that they will also escape eventually?

This argument also doesn't seem to be unique to mind uploading. Suppose that we achieved biological immortality and never uploaded. You could also make the argument that, now that people can live until the heat-death of the universe (or at least until our sun goes out), then their lifetimes are sufficiently long that at some point in their lives they are going to be kidnapped and tortured indefinitely by someone sadistic, so therefore we should kill everyone before we get radical life extension.

But for biological people, this argument doesn't feel anywhere near as compelling. In particular, this scenario highlights the fact that even though there might be a non-zero probability for any given person to be kidnapped and tortured during their lifetimes, that probability can be low enough that it's still unlikely to happen even during a very long lifetime.

You could reasonably argue that for uploads, it's different, since it's easier to make a copy of an upload undetected etc., so the probability of being captured during one's lifetime is larger. But note that there have been times in history that there actually was a reasonable chance for a biological human to be captured and enslaved during their lifetime! Back during the era of tribal warfare, for example. But we've come a long way from those times, and in large parts of the world, society has developed in such a way to almost eliminate that risk.

That, in turn, highlights the point that it's too simple to just look at whether we are biological or uploads. It all depends on how exactly society is set up, and how strong are the defenses and protections that society provides to the common person. Given that we've developed to the point where biological persons have pretty good defenses against being kidnapped and enslaved, to the point where we don't think that even a very long lifetime would be likely to lead to such a fate, shouldn't we also assume that upload societies could develop similar defenses and reduce the risk to be similarly small?

Comment author: RedMan 05 January 2018 01:32:59PM 1 point [-]

Thank you for the detailed response!

I agree that the argument you advance here is the sane one, but I have trouble reconciling it with my interpretation of Effective Altruism: 'effort should be made to expend resources on preventing suffering, maximize the ratio of suffering avoided to cost expended'

I interpret your paper as rejecting the argument advanced by prof Hansen that if of all future variants of you, the number enjoying 'heaven' vastly outnumber the copies suffering 'hell', on balance, uploading is a good. Based on your paper's citation of Omelas, I assert that you would weight 'all future heaven copies' in aggregate, and all future hell copies individually.

So if the probability of one or more hell copies of an upload coming into existence for as long as any heaven copy exceeds the probability of a single heaven copy existing long enough to outlast all the hell copies, that person's future suffering will eventually exceed all suffering previously experienced by biological humans. Under the EA philosophy described above, this creates a moral imperative to prevent that scenario, possibly with a blender.

If uploading tech takes the form of common connection and uploading to an 'overmind', this can go away--if everyone is Borg, there's no way for a non-Borg to put Borg into a hell copy, only Borg can do that to itself, which is, at least from an EA standpoint, probably an acceptable risk.

At the end of the day, I was hoping to adjust my understanding of EA axioms, not be talked down from chasing my friends around with a blender, but that isn't how things went down.

SF is a tolerant place, and EAs are sincere about having consistent beliefs, but I don't think my talk title "You helped someone avoid starvation with EA and a large grant. I prevented infinity genocides with a blender" would be accepted at the next convention.

Comment author: RedMan 05 January 2018 12:15:25AM 1 point [-]

Curious about your take on my question here: http://lesswrong.com/lw/os7/unethical_human_behavior_incentivised_by/ Awesome paper.

Comment author: Lumifer 04 January 2018 03:47:35PM 0 points [-]

Are you reinventing Asimov's Three Laws of Robotics?

Comment author: RedMan 04 January 2018 07:17:21PM *  0 points [-]

I hadn't thought about it that way.

I do think that either compiler time flags for the AI system or a second 'monitor' system chained to the AI system in order to enforce the named rules would probably limit the damage.

The broader point is that probabilistic AI safety is probably a much more tractable problem than absolute AI safety for a lot of reasons, to further the nuclear analogy, emergency shutdown is probably a viable safety measure for a lot of the plausible 'paperclip maximizer turns us into paperclips' scenarios.

"I need to disconnect the AI safety monitoring robot from my AI-enabled nanotoaster robot prototype because it keeps deactivating it" might still be the last words a human ever speaks, but hey, we tried.

Comment author: RedMan 04 January 2018 02:00:03AM *  0 points [-]

Rules for an AI:

If an action it takes results in more than N logs of $ worth of damage to humans/kills more than N logs of humans, transfer control of all systems it can provide control inputs to designated backup (human, formally proven safe algorithmic system, etc), power down.

When choosing among actions which affect a system external to it, calculate probable effect on human lives. If probability of exceeding N assigned in rule 1 is greater than some threshold Z, ignore that option, if no options are available, loop.

Most systems would be set to N= 1, Z = 1/10000, giving us five 9s of certainty that the AI won't kill anyone. Some systems (weapons, climate management, emergency management dispatch systems) will need higher N scores and lower Z scores to maintain effectiveness.

JFK had an N of like 9 and a Z score of 'something kind of high', and passed control to Lyndon B Johnson of 'I have a minibar and a shotgun in the car I keep on my farm so I can drive and shoot while intoxicated' fame. We survived that, we will be fine.

Are we done?

Comment author: Elo 03 January 2018 03:35:02PM 1 point [-]

Interesting you say that about bad when you are not lifting. There just wasn't any warning from anyone (there probably was but I took no notice).

I have been back to doctors, and I do run several times a week these days. It's not set wrong or else I couldn't run, and I never got an x-ray.

I went back to trampoline 6 months later and I injured myself trying to do something that I didn't have the muscles for any more. It strikes me as more dangerous than I was willing to admit. It's exercise that really pushes your body and I'm not sure I am comfortable with it compared to things that are more within a body's limit.

For example rock climbing - you are limited by what your body let's you do. Only lift your own weight. And that's a lot closer to the safe limit than trampolines which interact with external contraption and do things like compress your spine and cause unnatural brain shaking.

Weakest link theory was a bit of a joke but I am sure there is some truth to it.

Comment author: RedMan 04 January 2018 01:45:11AM 0 points [-]

That's great to hear, stay safe.

This sort of data was a contributor to my choice of sport for general well being: https://graphiq-stories.graphiq.com/stories/11438/sports-cause-injuries-high-school#Intro

There is truth to it: https://www.westside-barbell.com/blogs/2003-articles/extra-workouts-2

Really grateful for the info, I never could put my finger on what exactly I was not liking about CM when I wasn't pushing myself, stuff is amazing for preventing exercise soreness though.

Comment author: RedMan 03 January 2018 12:50:47PM 0 points [-]

You back to trampolining yet?

Way to eat a broken bone and not seek medical attention for it, someone I knew did about what you did and ended up having to have a doctor re-break and set the bone to fix things. Lots of 'newly fit' people, particularly teenagers, have your 'injury from stupidity' behavior pattern. This is one of the reasons professional athletes are banned from amateur sports by their contracts

The great coach Louis Simmons is worth reading, he will expand your mind on your weakest link theory.

My own conclusion on your magic enlightenment pill, based on my lived experience: super awesome when you're lifting, Fs you up a bit when you're not. Use it around intense exercise, otherwise avoid.

Comment author: Crab 28 October 2017 05:48:46AM 0 points [-]

Reading high quality works by the giants of the 20th century and comparing it to more modern material can be a humbling experience for some--it certainly has been for me on more than one occasion.

I am not sure, are you saying, that for some fields "works by the giants of the 20th century" is great, while modern material is bad?

Comment author: RedMan 21 December 2017 07:54:35PM *  0 points [-]

Correct. I have found that the works written at the time when the relevant technical work had just recently been completed, by the people who made those breakthroughs, is often vastly superior to summary work written decades after the field's last major breakthrough.

If I remember correctly, Elon Musk cited some older texts on rocketry as his 'tree trunk' of knowledge about the subject.

This advice only applies to mature fields, in places where fundamental breakthroughs are happening regularly, this advice is downright awful.

Comment author: turchin 05 December 2017 03:36:27PM 0 points [-]

What if we fail to make them in time, and instead fooming non-human AI appear?

Comment author: RedMan 21 December 2017 07:49:15PM 0 points [-]

Assertion: Any fooming non-human AI incompatible with uplifting technology would be too alien to communicate with in any way. If you happen to see one of those, probably a good idea to just destroy it on sight.

Comment author: RedMan 05 December 2017 02:34:04AM *  0 points [-]

If we can build uplifted human-ai hybrids, which eventually will be more engineered than evolved, why bother building any more baseline humans?

View more: Next