= 27d567bc2d48d9f40e9ccc20ea370b15)
Strange7 comments on Hedging our Bets: The Case for Pursuing Whole Brain Emulation to Safeguard Humanity's Future - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
Comments (244)
What about those of us who would prefer indefinite human-directed torture to instantaneous cessation of existence? I have no personal plans to explore masochism in that sort of depth, particularly in a context without the generally-accepted safety measures, but it's not the worst thing I can imagine. I'd find ways to deal with it, in the same sense that if I were stranded on a desert island I would be more willing to gag down whatever noxious variety of canned fermented fish was available, and eventually learn to like it, rather than starve to death.
I don't think you are appreciating the potential torture that could be inflicted by a superintelligence dedicated to advancing anti-fun theory. Such a thing would likely make your mind bigger at some optimal rate just so you could appreciate the stream of innovative varieties of enormous pain (not necessarily normal physical pain) it causes you.
You can't imagine torture that is worse than death?
By 'death' I assume you mean the usual process of organ failure, tissue necrosis, having what's left of me dressed up and put in a fancy box, followed by chemical preservation, decomposition, and/or cremation? Considering the long-term recovery prospects, no, I don't think I can imagine a form of torture worse than that, except perhaps dragging it out over a longer period of time or otherwise embellishing on it somehow.
This may be a simple matter of differing personal preferences. Could you please specify some form of torture, real or imagined, which you would consider worse than death?
Suppose I was tortured until I wanted to die. Would that count?
There have been people who wanted to die for one reason or another, or claimed to at the time with apparent sincerity, and yet went on to achieve useful or at least interesting things. The same cannot be said of those who actually did die.
Actual death constitutes a more lasting type of harm than anything I've heard described as torture.
There's a nihilism lurking here which seems at odds with your unconditional affirmation of life as better than death. You doubt that anything anyone has ever done was "useful"? How do you define useful?
Admittedly, my personal definition isn't particularly rigorous. An invention or achievement is useful if it makes other people more able to accomplish their existing goals, or maybe if it gives them something to do when they'd otherwise be bored. It's interesting (but not necessarily useful) if it makes people happy, is regarded as having artistic value, etc.
Relevant examples: Emperor Norton's peaceful dispersal of a race riot was useful. His proposal to construct a suspension bridge across San Francisco Bay would have been useful, had it been carried out. Sylvia Plath's work is less obviously useful, but definitely interesting.
Most versions of torture, continued for your entire existence. You finally cease when you otherwise would (at the heat death of the universe, if nothing else), but your entire experience spent being tortured. The type isn't really important, at that point.
First, the scenario you describe explicitly includes death, and as such falls under the 'embellishments' exception.
Second, thanks to the hedonic treadmill, any randomly-selected form of torture repeated indefinitely would eventually become tolerable, then boring. As you said,
Third, if I ever run out of other active goals to pursue, I could always fall back on "defeat/destroy the eternal tormetor of all mankind." Even with negligible chance of success, some genuinely heroic quest like that makes for a far better waste of my time and resources than, say, lottery tickets.
What if your hedonic treadmill were disabled, or bypassed by something like direct stimulation of your pain center?
You're going to die (or at least cease) eventually, unless our understanding of physics changes significantly. Eventually, you'll run out of negentropy to run your thoughts. My scenario only changes what happens between then and now.
Failing that, you can just be tortured eternally, with no chance of escape (no chance of escape is unphysical, but so is no chance of death). Even if the torture becomes boring (and there may be ways around that), an eternity of boredom, with no chance to succeed any at any goal, seems worse than death to me.
When considering the potential harm you could suffer from a superintelligence that values harming you, you don't get to exclude some approaches it could take because they are too obvious. Superintelligences take obvious wins.
Perhaps. So consider other approaches the hostile superintelligence might take. It's not going to go easy on you.
Yes, I've considered the possibility of things like inducement of anteriograde amnesia combined with application of procedure 110-Montauk, and done my best to consider nameless horrors beyond even that.
As I understand it, a superintelligence derived from a sadistic, sociopathic human upload would have some interest in me as a person capable of suffering, while a superintelligence with strictly artificial psychology and goals would more likely be interested in me as a potential resource, a poorly-defended pile of damp organic chemistry. Neither of those is anywhere near my ideal outcome, of course, but in the former, I'll almost certainly be kept alive for some perceptible length of time. As far as I'm concerned, while I'm dead, my utility function is stuck at 0, but while I'm alive my utility function is equal to or greater than zero.
Furthermore, even a nigh-omnipotent sociopath might be persuaded to torture on a strictly consensual basis by appealing to exploitable weaknesses in the legacy software. The same cannot be said of a superintelligence deliberately constructed without such security flaws, or one which wipes out humanity before it's flaws can be discovered.
Neither of these options is actually good, but the human-upload 'bad end' is at least, from my perspective, less bad. That's all I'm asserting.
Yes, the superintelligence that takes an interest in harming you would have to come from some optimized process, like recursive self improvement of a psychopath upload.
A sufficient condition for the superintelligence to be indifferent to your well being, and see you as spare parts, is an under optimized utility function.
Your approach to predicting what the hostile superintelligence would do to you, seems to be figuring out the worst sort of torture that you can imagine. The problem with this is that the superintelligence is a lot smarter, and more creative than you. Reading your mind and making real you worst fears, constantly with no break or rest, isn't nearly as bad as what it would come up with. And no, you are not going to find some security flaw you can exploit to defeat it, or even slow it down. For one thing, the only way you will be able to think straight is if it determines that this maximizes the harm you experience. But the big reason is recursive self improvement. The superintelligence will analyze itself and fix security holes. You, puny mortal, will be up against a superintelligence. You will not win.
If you knew you were going to die tomorrow, would you now have a preference for what happens to the universe afterwards?
A superintelligence based on an uploaded human mind might retain exploits like 'pre-existing honorable agreements' or even 'mercy' because it considers them part of it's own essential personality. Recursive self-improvement doesn't just mean punching some magical enhance button exponentially fast.
My preferences would be less relevant, given the limited time and resources I'd have with which to act on them. They wouldn't be significantly changed, though. I would, in short, want the universe to continue containing nice places for myself and those people I love to live in, and for as many of us as possible to continue living in such places. I would also hope that I was wrong about my own imminent demise, or at least the inevitability thereof.
If we are postulating a superintelligence that values harming you, let's really postulate that. In the early phases of recursive self improvement, it will figure out all the principles of rationality we have discussed here, including the representation of preferences as a utility function. It will self-modify to maximize a utility function that best represents its precursor conflicting desires, including hurting others and mercy. If it truly started as a psychopath, the desire to hurt others is going to dominate. As it becomes superintelligent, it will move away from having a conflicting sea of emotions that could be manipulated by someone at your level.
I was never suggesting it was anything magical. Software security, given physical security of the system, really is not that hard. The reason we have security holes in computer software today is that most programmers, and the people they work for, do not care about security. But a self improving intelligence will at some point learn to care about its software level security (as an instrumental value), and it will fix vulnerabilities in its next modification.
Is it fair to say that you prefer A: you die tomorrow and the people you currently care about will continue to have worthwhile lives and survive to a positive singularity, to B: you die tomorrow and the people you currently care about also die tomorrow?
If yes, then "while I'm dead, my utility function is stuck at 0" is not a good representation of your preferences.
It is truly astonishing how much pain someone can learn to bear - AdeleneDawner posted some relevant links a while ago.
Edit: I wasn't considering an anti-fun agent, however - just plain vanilla suffering.