Review
Saying the quiet part out loud: trading off x-risk for personal immortality
57interstice
14Adele Lopez
4AlphaAndOmega
3O O
12disturbance
15mako yass
3lc
1O O
1Dirichlet-to-Neumann
1O O
2frontier64
3interstice
2frontier64
5interstice
45Ben Pace
24Matthew Barnett
23Ben Pace
10Matthew Barnett
23Ben Pace
7Matthew Barnett
0Ben Pace
8Matthew Barnett
6lc
6Matthew Barnett
2lc
5Matthew Barnett
2lc
6Matthew Barnett
2lc
2RHollerith
12Matthew Barnett
6Valentine
24Odd anon
4disturbance
22DanielFilan
1Shankar Sivarajan
4Oliver Sourbut
17Kenoubi
6Lyrialtus
4AlphaAndOmega
3AnthonyC
3Matt Goldenberg
1AlphaAndOmega
2Matt Goldenberg
1AlphaAndOmega
2Kenoubi
2AlphaAndOmega
1Kenoubi
15Oliver Sourbut
10O O
3Oliver Sourbut
4O O
2Dirichlet-to-Neumann
5O O
-14Shankar Sivarajan
-1Oliver Sourbut
15Algon
2O O
14ImmortalityOrDeathByAGI
7ImmortalityOrDeathByAGI
1Alex K. Chen (parrot)
6avturchin
6Insub
-4avturchin
2Insub
1frontier64
5lc
1disturbance
10lc
4Oliver Sourbut
5niki.h
4Nathaniel Monson
2Dagon
4npostavs
4pathos_bot
2AnthonyC
2RHollerith
2Dagon
1Oliver Sourbut
1Shankar Sivarajan
1Sinclair Chen
1frontier64
1simple_name
1Ixiel
-2[deactivated]
-2Oliver Sourbut
-2Ann
-3nim
-6lenivchick
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)
At this point timelines look short enough that you likely increase your personal odds of survival more by increasing the chance that AI goes well than by speeding up timelines. Also I don't see why you think cryonics doesn't make sense as alternative option.
4
I wish I could convince my grandpa to sign up for cryonics, but he's a 95 yo Indian doctor in India, where facilities for cryopreservation only extends to organs and eggs, so it's moot regardless of the fact that I can't convince him.
I expect my parents to survive to the Singularity, whether or not it kills us in the process. Same for me, and given my limited income, I'm not spending it on cryonics given that a hostile AGI will kill even the ones frozen away.
3
You probably also irreversibly lose a ton of information with cryonics.
- The timelines certainly still looked short enough a couple of months ago. But what prompted me to write this was the 13th observation: the seemingly snowballing Pause movement, which, once it reaches certain threshold, has a potential to significantly stifle the development of AI. Analogies: human genetic enhancement, nuclear energy. I'm not sure whether this is already past the point of countering the opposite forces (useful applications, Moore's law), but I'm also not sure that it isn't (or won't be soon).
- Cryonics is a very speculative tech. We don't understand how much information is lost in the process, scientific evidence seems lacking overall - consensus being it's in the ~few percent success probability region, future AI (future society) would have to want to revive humans instead of creating new ones, etc.
> consensus being it's in the ~few percent success probability region
Consensus among who? I haven't been able to find a class of experts I'd defer to. We have Alcor, who are too partial, we have the society of cryobiology who openly refuse to learn anything about the process and threaten to exile any member who does, and I have random members of the rationalist community who have no obligation to be right and just want to sound grounded.
3
In order for a pause to work it has to happen everywhere. Nuclear power is widely deployed in e.g. France, so you need a stronger political force than the one causing nuclear power not to proliferate.
1
AI is also more like the “keys to the kingdom” here.
The benefit of nuclear power isn’t that huge, you still have fossil fuels which are cheap (even if they cause climate change in the long run).
The benefits of genetic editing/eugenics is also pretty nebulous and may take decades to realize.
On the other hand, one country having an aligned ASI offers an overwhelming advantage. World dominance goes from fiction to mundane reality. I think these sort of treaties also advertise this fact so I think it’s likely they won’t work. All governments are probably seriously considering what I mentioned above.
Why is the U.S. blocking Chinas access to GPUs? That seems like the most plausible explanation.
1
High levels GPUs are needed for basically anything mundane today. No need to bring in AGI worries to make it a strategic ressource.
1
I think the timing and perf focus of it all makes it clear it’s related to foundation models.
2
If you think cryonics has a very high likelihood of working then sure. I don't think the arguments that cryonics is likely to work are that good though. I don't think Eliezer has even made arguments to that effect. They were mostly "hey doing cryonics is better than not doing it because not doing cryonics is just death!"
3
I think Eliezer is pretty confident that cryonics will work. For myself, I'm not sure, haven't really looked into it that deeply, but the a priori argument makes sense and I feel like this is the kind of thing people would be irrationally biased against due to its speculative nature(similar to the AGI skepticism that many had until recently) so I'd give it decent odds.
2
I was responding to this point. The "cryonics is better than nothing" argument doesn't make cryonics an alternative option to immortality by friendly AI. If Bob thinks cryonics has a 10% chance of making him immortal and thinks AI will have a 20% chance of making him immortal and an 80% chance of destroying the world, then the superhuman AI route is more likely to lead to Bob's immortality than cryonics.
5
I didn't say that "cryonics is better than nothing", I said I think it has decent odds of success. To spell it out, I think the success probability is higher than the increased probability of friendly AI in my lifetime from acceleration(which is the relevant comparison) while imposing fewer costs on future generations. And I think that if you made it your life's work, you could probably improve those odds, up to 80% perhaps(conditional on future people wanting to revive you)
I do not want the timing of AGI development to be based on my personal life-expectancy. Going further I think most decent people would be willing to sacrifice their own life to prevent their civilization from going extinct, and I think it would be a very honorable thing to do [1]. Every single generation to come is at stake, so I don't think my own life bears much on whether to defend all of theirs.
- ^
Not that I am aware of any such opportunity, to be clear.
Every single generation to come is at stake, so I don't think my own life bears much on whether to defend all of theirs.
Note that the idea that >10^50 lives are at stake is typically premised on the notion that there will be a value lock-in event, after which we will successfully colonize the reachable universe. If there is no value lock-in event, then even if we solve AI alignment, values will drift in the long-term, and the stars will eventually be colonized by something that does not share our values. From this perspective, success in AI alignment would merely delay the arrival of a regime of alien values, rather than prevent it entirely. If true, this would imply that positive interventions now are not as astronomically valuable as you might have otherwise thought.
My guess is that the idea of a value lock-in sounded more plausible back in the days when people were more confident there will be (1) a single unified AI that takes control over the world with effectively no real competition forever, and (2) this AI will have an explicit, global utility function over the whole universe that remains unchanging permanently. However, both of these assumptions seem dub...
The relevance of value drift here sounds to me analogous to the following exchange.
Alice, who is 10 years old: "Ah, I am about to be hit by a car, I should make sure this doesn't happen else I'll never get to live the rest of my life!"
Bob: "But have you considered that you probably won't really be the same person in 20 years? You'll change what you value and where you live and what your career is and all sorts. So I wouldn't say your life is at stake right now."
The concern about AI misalignment is itself a concern about value drift. People are worried that near-term AIs will not share our values. The point I'm making is that even if we solve this problem for the first generation of smarter-than-human AIs, that doesn't guarantee that AIs will permanently share our values in every subsequent generation. In your analogy, a large change in the status quo (death) is compared to an arguably smaller and more acceptable change over the long term (biological development). By contrast, I'm comparing a very bad thing to another similarly very bad thing. This analogy seems mostly valid only to the extent you reject the premise that extreme value drift is plausible in the long-term, and I'm not sure why you would reject that premise.
1Putting "cultural change" and "an alien species comes along and murders us into extinction" into the same bucket seems like a mistake to me. I understand that in each literally one set of values replaces another. But in the latter case, the route by which those values change is something like "an alien was grown in order to maximize a bunch of functions that we were able to define like stock price and next-token prediction and eventually overthrew us", and I think that is qualitatively different than "people got richer and wealthier and so what they wanted changed" in a way that is likely to be ~worthless from our perspective.
From reading elsewhere my current model (which you may falsify) is that you think that those values will be sufficiently close that they will still be very valuable from our perspective, or about as valuable as people from 2000 years ago would think us today. I don't buy the first claim; I think the second claim is more interesting but I'm not really confident that it's relevant or true.
(Consider this an open offer to dialogue about this point sometime, perhaps at my dialogues party this weekend.)
7
Value drift encompasses a lot more than cultural change. If you think humans messing up on alignment could mean something as dramatic as "an alien species comes along and murders us", surely you should think that the future could continue to include even more, similarly dramatic shifts. Why would we assume that once we solve value alignment for the first generation of AIs, values would then be locked in perfectly forever for all subsequent generations?
0
8
I don't understand how this new analogy is supposed to apply to the argument, but if I wanted to modify the analogy to get my point across, I'd make Alice 90 years old. Then, I'd point out that, at such an age, getting hit by a car and dying painlessly genuinely isn't extremely bad, since the alternative is to face death within the next several years with high probability anyway.
6
If you actually believe that at some point, even with aligned AI, the forces of value drift are so powerful that we are still unlikely to survive, then you've just shifted the problem to avoiding this underspecified secondary catastrophe in addition to the first. The astronomical loss is still there.
6
Human survival is different from the universe being colonized under the direction of human values. Humans could survive, for example, as a tiny pocket of a cosmic civilization.
Indeed, but the question is, "What was the counterfactual?" If solving AI alignment merely delays an astronomical loss, then it is not astronomically important to solve the problem. (It could still be very important in this case, but just not so important that we should think of it as saving 10^50 lives.)
2
Can you specify the problem(s) you think we might need to solve, in addition to alignment, in order to avoid this sterility outcome?
5
We would need to solve the general problem of avoiding value drift. Value drift is the phenomenon of changing cultural, social, personal, biological and political values over time. We have observed it in human history during every generation. Older generations vary on average in what they want and care about compared to younger generations. More broadly, species have evolved over time, with constant change on Earth as a result of competition, variation, and natural selection. While over short periods of time, value drift tends to look small, over long periods of time, it can seem enormous.
I don't know what a good solution to this problem would look like, and some proposed solutions -- such as a permanent, very strict global regime of coordination to prevent cultural, biological, and artificial evolution -- might be worse than the disease they aim to cure. However, without a solution, our distant descendants will likely be very different from us in ways that we consider morally relevant.
2
How do you expect this value drift to occur in an environment where humans don't actually have competitive mental/physical capital? Presumably if humans "drift" beyond some desired bounds, the coalition of people or AIs with the actual compute and martial superiority who are allowing the humans to have these social and political fights will intervene. It wouldn't matter if there was one AI or twenty, because the N different optimizers are still around and optimizing for the same thing, and the equilibrium between them wouldn't be meaningfully shifted by culture wars between the newly formed humans.
6
Value drift happens in almost any environment in which there is variation and selection among entities in the world over time. I'm mostly just saying that things will likely continue to change continuously over the long-term, and a big part of that is that the behavioral tendencies, desires and physical makeup of the relevant entities in the world will continue to evolve too, absent some strong countervailing force to prevent that. This feature of our world does not require that humans continue to have competitive mental and physical capital. On the broadest level, the change I'm referring to took place before humans ever existed.
2
I do not understand how you expect this general tendency to continue against the will of one or more nearby gpu clusters.
2
Some people enjoy arguing philosophical points, and there is nothing wrong with that.
Do you believe that the considerations you have just described have any practical relevance to someone who believes that the probability of AI research's ending all human life some time in the next 60 years is .95 and wants to make a career out of pessimizing that probability?
Yes, I believe this point has practical relevance. If what I'm saying is true, then I do not believe that solving AI alignment has astronomical value (in the sense of saving 10^50 lives). If solving AI alignment does not have astronomical counterfactual value, then its value becomes more comparable to the value of other positive outcomes, like curing aging for people who currently exist. This poses a challenge for those who claim that delaying AI is obviously for the greater good as long as it increases the chance of successful alignment, since that could also cause billions of currently existing people to die.
6
While I agree, I want to take the opportunity to poke at something I often see in models like this one.
I think if you ask most people about this choice, they'd answer like you predict.
I think if you gave people a choice of buttons to push, one of which is this self-sacrifice button and the other being… uh… whatever the alternative is, there'd be some more hesitance but maybe not a ton.
But I suspect most people do not in fact understand what it means to die, or what they're agreeing to when they agree to sacrifice themselves for something.
I think this is precisely what changes when someone gets a terminal diagnosis. They come to understand in a lived way what it means that they, right there, inside that body and seeing from behind those eyes, are going to experience death. Their mortality isn't an abstraction for them anymore. They stop thinking "I'm going to die someday" like they're talking about a video came character and instead get that it means something way way deeper.
If you adjust the situation so that the person believes the argument that they need to die for the sake of their civilization, and then you hand them a gun with which to shoot themselves…
…I think you'd find the capacity to be "honorable" here dropping dramatically.
But not people saying they would shoot themselves in this hypothetical scenario! Because for the person thinking the thought experiment, the thought experiment doesn't embed the thought-experimenter.
I don't think this bears on the discussion of immortality vs. AI risk. The actions here are abstract enough to be more like the button-pushing case.
I just keep seeing this question of embedded agency getting skipped over and I think it's deeply important.
I've seen this kind of opinion before (on Twitter, and maybe reddit?), and I strongly suspect that the average person would react with extreme revulsion to it. It most closely resembles "cartoon villain morality", in being a direct tradeoff between everyone's lives and someone's immortality. People strongly value the possibility of their children and grandchildren being able to have further children of their own, and for things in the world to continue on. And of course, the statement plays so well into stereotypes of politically-oriented age differences: Old people not sufficiently caring about what happens after they die, so they'll take decisions that let young people deal with catastrophes, young people thinking they'll never die and being so selfish that they discount the broader world outside themselves, etc. If anything, this is a "please speak directly into the microphone" situation, where the framing would pull people very strongly in the direction of stopping AGI.
4
I think the current situation is/was greatly distorted by signalling games that people play. Once everyone realises that this is an actual choice, there is a chance they change their opinions to reflect the true tradeoff. (This depends a lot on network effects, shifting Overton window etc., I'm not claiming that 100% of the effect would be rational consideration. But I think rational consideration biases the process to in a non-negligible way.). But yes, one of the pieces of evidence is how old people don't seem to particularly care about the future of civilisation.
But yes, one of the pieces of evidence is how old people don't seem to particularly care about the future of civilisation.
I don't think this is true. Stereotype evidence: your average conservative old person loves their country and is mad that college campuses are liberal because it means elites will ruin it.
1
Everyone's immortality. They don't typically make cartoon villains like that.
4
The framing of OP is specifically about racing just slowly enough that some specific people make the cut[1], which is absolutely a tradeoff between everyone's lives and those specific people's immortality. OP is explicitly indifferent about the effects on other people, including those who die sooner and those whose chances might be ruined by moving too fast for safety.
----------------------------------------
1. which rests on a possibly-questionable received wisdom that AGI is a sufficient and necessary route to immediate immortality ↩︎
I agree with the Statement. As strongly as I can agree with anything. I think the hope of current humans achieving... if not immortality, then very substantially increased longevity... without AI doing the work for us, is at most a rounding error. And ASI that was even close to aligned, that found it worth reserving even a billionth part of the value of the universe for humans, would treat this as the obvious most urgent problem and solve death pretty much if there's any physically possible way of doing so. And when I look inside, I find that I simply don't care about a glorious transhumanist future that doesn't include me or any of the particular other humans I care about. I do somewhat prefer being kind / helpful / benificent to people I've never met, very slightly prefer that even for people who don't exist yet, but it's far too weak a preference to trade off against any noticeable change to the odds of me and everyone I care about dying. If that makes me a "sociopath" in the view of someone or other, oh well.
I've been a supporter of MIRI, AI alignment, etc. for a long time, not because I share that much with EA in terms of values, but because the path to the future having an...
6
Thank you for writing this. I usually struggle to find resonating thoughts, but this indeed resonates. Not all of it, but many key points have a reflection that I'm going to share:
* Biological immortality (radical life extension) without ASI (and reasonably soon) looks hardly achievable. It's a difficult topic, but for me even Michael Levin's talks are not inspiring enough. (I would rather prefer to become a substrate-independent mind, but, again, imagine all the R&D without substantial super-human help.)
* I'm a rational egoist (more or less), so I want to see the future and have pretty much nothing to say about the world without me. Enjoying not being alone on the planet is just a personal preference. (I mean, the origin system is good, nice planets and stuff, but what if I want to GTFO?) Also, I don't trust imaginary agents (gods, evolution, future generations, AGIs), however creating some of them may be rational.
* Let's say that early Yudkowsky has influenced my transhumanist views. To be honest, I feel somewhat betrayed. Here my position is close to what Max More says. Basically, I value the opportunities, even if I don't like all the risks.
* I agree that AI progress is really hard to stop. The scaling leaves possible algorithmic breakthroughs underexplored. There is so much to be done, I believe. The tech world will still be working on it even with mediocre hardware. So we are going to ASI anyway.
* And all the alignment plans... Well, yes, they tend to be questionable. For me, creating human-like agency in AI (to negotiate with) is more about capabilities, but that's a different story.
4
I respectfully disagree on the first point. I am a doctor myself and given observable increase in investment in life extension (largely in well funded stealth startups or Google Calico), I have ~70% confidence that in the absence of superhuman AGI or other x-risks in the near term, we have a shot at getting to longevity escape velocity in 20 years.
While my p(doom) for AGI is about 30% now, down from a peak of 70% maybe 2 years ago after the demonstration that it didn't take complex or abstruse techniques to reasonably align our best AI (LLMs), I can't fully endorse acceleration on that front because I expect the tradeoff in life expectancy to be net negative.
YMMV, it's not like I'm overly confident myself at 70% for life expectancy being uncapped, and it's not like we're probably going to find out either. It just doesn't look like a fundamentally intractable problem in isolation.
3
I also see multiple technological pathways that would get us to longevity escape velocity that seem plausible without AGI in that timeframe.
If nothing else, with advances in tissue engineering I expect we will be able to regrow and replace every organ in the body except the brain by mid-century.
But I also think a lot of what's needed is culturally/politically/legally fraught in various ways. I think if we don't get life-extending tech, it will be because we made rules inadvertently preventing it or pretending it's better not to have it.
3
Is the claim here a 70% chance of longevity escape velocity by 2043? It's a bit hard to parse.
If that is indeed the claim, I find it very surprising, and I'm curious about what evidence you're using to make that claim? Also, is that LEV for like, a billionaire, a middle class person in a developed nation, or everyone?
1
Yes, you can reformat it in that form if you prefer.
This is a gestalt impression based off my best impressions of the pace of ongoing research (significantly ramped up compared to where investment was 20 years ago), human neurology, synthetic organs and finally non-biological alternatives like cybernetic enhancement. I will emphasize that LEV != actual biological immortality, but it leads to at least a cure for aging if nothing else.
Aging, while complicated and likely multifactorial, doesn't seem intractable to analysis or mitigation. We have independent research projects tackling individual aspects, but as I've stated, most of them are in stealth mode even if they're well-funded, and solving any individual mechanism is insufficient because of how aging itself is an exponential process.
To help, I'm going to tackle the top causes of aging in the West-
1. Heart disease- This is highly amenable to outright replacement of the organ, be it with a cybernetic replacement or one grown in-vitro. Obesity, which contributes heavily to cardiovascular disease and morbidity, is already being tackled by the discovery of GLP-1 antagonists like semaglutide, and I fully expect that the obesity epidemic that is dragging down life expectancy in the West will be over well before then.
2. Cancer- Another reason for optimism, CAR-T therapy is incredibly promising, as are other targeted therapies. So are vaccines for diseases like HPV that themselves cause cancer (said vaccine already exists, I'm talking more generally).
3. Unintentional injuries- The world has grown grossly safer, and only will continue to do so, especially as things get more automated.
4. Respiratory diseases- Once again, reason for optimism that biological replacements will be cheap enough that we won't have to rely on limited numbers of donors for transplants.
5. Stroke and cerebrovascular disease- I'll discuss the brain separately, but while this is a harder subject to tackle, mitigating obesity helps
2
You, being a relatively wealthy person in a modernized country? Do you think you'll be able to afford the LEV by that time, or only that some of the wealthiest people will?
1
I'm a doctor in India right now, and will likely be a doctor in the UK by then, assuming I'm not economically obsolete. And yes, I expect that if we do have therapies that help provide LEV, they will be affordable in my specific circumstances as well as most LW readers, if not globally. UK doctors are far poorer compared to the their US kin.
Most biological therapies are relatively amenable to economies of scale, and while there are others that might be too bespoke to manage the same, that won't last indefinitely. I can't imagine anything with as much demand as a therapy that is proven to delay aging nigh indefinitely, for an illustrative example look at what Ozempic and Co are achieving already, every pharma industry leader and their dog wants to get in on the action, and the prices will keep dropping for a good while.
It might even make economic sense for countries to subsidize the treatment (IIRC, it wouldn't take much more for GLP-1 drugs to reach the point where they're a net savings for insurers or governments in terms of reducing obesity related health expenditures). After all, aging is why we end up succumbing to so many diseases in our senescence, not the reverse.
Specifically, gene therapy will likely be the best bet for scaling, if a simple drug doesn't come about (seems unlikely to me, I doubt there's such low hanging fruit, even if the net result of LEV might rely on multiple different treatments in parallel with none achieving it by themself).
2
Thanks for your reply. "70% confidence that... we have a shot" is slightly ambiguous - I'd say that most shots one has are missed, but I'm guessing that isn't what you meant, and that you instead meant 70% chance of success.
70% feels way too high to me, but I do find it quite plausible that calling it a rounding error is wrong. However, with a 20 year timeline, a lot of people I care about will almost definitely still die, who could have not died if death were Solved, which group with very much not negligible probability includes myself. And as you note downthread, the brain is a really deep problem with prosaic life extension. Overall I don't see how anything along these lines can be fast enough and certain enough to be a crux on AI for me, but I'm glad people are working on it more than is immediately apparent to the casual observer. (I'm a type 1 diabetic and would have died at 8 years old if I'd lived before insulin was discovered and made medically available, so the value of prosaic life extension is very much not lost on me.)
2
T1DM is a nasty disease, and much like you, I'm more than glad to live in the present day when we have tools to tackle it, even if other diseases still persist. There's no other time I'd rather be alive, even if I die soon, it's going to be interesting, and we'll either solve ~all our problems or die trying.
I understand. My mother has chronic liver disease, and my grandpa is 95 years old, even if he's healthy for his age (a low bar!). In the former case, I think she has a decent chance of making it to 2043 in the absence of a Singularity, even if it's not as high as I would like. As for my grandfather, at that age just living to see the next birthday quickly becomes something you can't take for granted. I certainly cherish all the time I can spend him with him, and hope it all goes favorably for us all.
As for me, I went from envying the very young, because I thought they were shoe-ins for making it to biological immortality, to pitying them more these days, because they haven't had at least the quarter decade of life I've had in the event AGI turns out malign.
Hey, at least I'm glad we're not in the Worst Possible Timeline, given that awareness of AI x-risk has gone mainstream. That has to count for something.
1
P.S. Having this set of values and beliefs is very hard on one's epistemics. I think it's a writ-large version of what Eliezer has stated as "thinking about AI timelines is bad for one's epistemics". Here are some examples:
(1) Although I've never been at all tempted by e/acc techno-optimism (on this topic specifically) / alignment isn't a problem at all / alignment by default, boy, it sure would be nice to hear about a strategy for alignment that didn't sound almost definitely doomed for one reason or another. Even though Eliezer can (accurately, IMO) shoot down a couple of new alignment strategies before getting out of bed in the morning. So far I've never found myself actually doing it, but it's impossible not to notice that if I just weren't as good at finding problems or as willing to acknowledge problems found by others, then some alignment strategies I've seen might have looked non-doomed, at least at first...
(2) I don't expect any kind of deliberate slowdown of making AGI to be all that effective even on its own terms, with the single exception of indiscriminate "tear it all down", which I think is unlikely to get within the Overton window, at least in a robust way that would stop development even in countries that don't agree (forcing someone to sabotage / invade / bomb them). Although such actions might buy us a few years, it seems overdetermined to me that they still leave us doomed, and in fact they appear to cut away some of the actually-helpful options that might otherwise be available (the current crop of companies attempting to develop AGI definitely aren't the least concerned with existential risk of all actors who'd develop AGI if they could, for one thing). Compute thresholds of any kind, in particular, I expect to lead to much greater focus on doing more with the same compute resources rather than doing more by using more compute resources, and I expect there's a lot of low-hanging fruit there since that isn't where people have been focusing,
Apparent moral alien here. Hi! I'm pretty astonished how many people apparently believe something like that statement. In my world, it's an off-the-deep-end perverse and sociopathic sort of value to hold. Of course, it's consistent. I can work with sociopaths, because to a first approximation I can work with anyone. But if there's enough delay, cooperation is going to become strained, because I'm quite happy with me dying, and you dying, and your loved ones dying (ideally painlessly having lived fulfilling lives!), if it turns out to be necessary in order for there to be a valuable future at all.
Despite introducing myself as an alien, I currently think most humans don't espouse your statement, because most humans' moral circle is big enough to preclude it. Other humans reject it on the basis of philosophical open/empty individualism. Others reject it for superstitious reasons or due to ethical injunctions. Others still for the deceptive reasons you mentioned. There are basically a lot of reasons either not buy it, or be quiet about it if you do!
As an aside, it appears that in certain circles, the deceptive/prudent reasons you mention to not espouse this view are inverted: a sort ...
1
1Many humans, given a choice between
A) they and their loved ones (actually everyone on earth) will live forever with an X-risk p
B) this happens after they and everyone they love dies with an X-risk less than p
Would choose A.
Abortion has a sort of similar parallel but with economic risk instead of X risk, and obviously no immortality yet many are pro choice.
I think valuing the lives of future humans you don’t know of over the lives of yourselves and your loved ones is the alien choice here.
3
I think any human with time-consistent preferences prefers A to B for some margin? The question is how much margin.
Don't understand your abortion analogy at all I'm afraid.
I did introduce myself as 'apparent moral alien'! - though for the reasons in the rest of my comment I don't think I'm all that rare. Until quite recently I'd have been confident I was in a supermajority, but I'm less sure of that now, and I weakly entertain the hypothesis it's a minority.
4
You are ending a potential future life in exchange for preserving a quality of present life.
2
Yet human routinely sacrifice their own lives for the good of other (see : firefighters, soldiers, high mountain emergency rescuers, etc.). The X-risk argument is more abstract but basically the same.
5
A lot of our moral frameworks breakdown once immortality is a real choice. Sacrificing your own life for the own good can be reframed as dying a little earlier.
Many of these people go in knowing there’s a small chance of death. A lot of them would probably change their minds if it was a suicide mission (except a minority).
If the 5 year mortality rate of firefighting was 100%, how many would still do that job?
-14
I kind of agree with the statement, but I don't quite agree with the implications: my timelines are already quite short and I think that if we contract things more, we pay a high cost in risk for little benifit unless you expect yourself and many of your loved ones to die before then. In which case, what's the point of discussing it?
Now if timelines were long, then yeah, it would make sense to accelerate AI insofar as it contributed to the continued surival of me and my loved ones. That could be through creating a FAI, or it could be via improved longevity tech, better disease treatment, improved cryonics, mind uploading etc. Which many people in this community are quite in favour of, even now.
Fifth observation: Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.
I think you underestimate how weird people find the desire for immortality. Why does pretty much no-one sign up for cryonics? Because it is unlikely to work? Nah, the odd's are maybe a few percent for current tech. Plenty of people should be willing to take a bet for continued life at those odds. Is it because of cost? Again, no. Life insurance isn't that expensive. The real reason is that it is weird.
2
It’s weird because it’s not what anyone else thinks of immortality. Freezing your corpse isn’t the same as being young forever in the present.
In reality, humans have been pondering immortality since time immemorial. A lot of our myths involve immortals and some humans becoming immortals. A lot of our religions assume there’s an immortal afterlife.
I wrote about this in Appendix A of this post.
------
One might look at the rough 50/50 chance at immortality given surviving AGI and think "Wow, I should really speed up AGI so I can make it in time!". But the action space is more something like:
- Work on AI safety (transfers probability mass from "die from AGI" to "survive AGI")
- The amount of probability transferred is probably at least a few microdooms per person.
- Live healthy and don't do dangerous things (transfers probability mass from "die before AGI" to "survive until AGI")
- Intuitively, I'm guessing one can transfer around 1 percentage point of probability by doing this.
- Do nothing (leaves probability distribution the same)
- Preserve your brain if you die before AGI (kind of transfers probability mass from "die before AGI" to "survive until AGI")
- This is a weird edge case in the model and it conditions on various beliefs about preservation technology and whether being "revived" is possible
- Delay AGI (transfers probability from "die from AGI" to "survive AGI" and from "survive until AGI" to "die before AGI")
- Accelerate AGI (transfer probability mass from "survive AGI" to "die from AGI" and from "die before AGI" to "survive until AGI")
I t...
7
For most people in their 20s or 30s, it is quite unlikely (around 10%) that they die before AGI. And if you basically place any value on the lives of people other than yourself, then the positive externalities of working on AI safety probably strongly outweigh anything else you could be doing.
Acceleration probably only makes sense for people who are (1) extremely selfish (value their life more than everything else combined) and (2) likely to die before AGI unless it's accelerated.
1
"10% is overconfident", given huge uncertainty over AGI takeoff (especially the geopolitical landscape of it), and especially given the probability that AGI development may be somehow slowed (https://twitter.com/jachaseyoung/status/1723325057056010680 )
Most longevity researchers will still be super-skeptical if you say AGI is going to solve LEV in our lifetimes (one could say - a la Structure of Scientific Revolutions logic - that most of them have a blindspot for recent AGI progress - but AGI=>LEV is still handwavy logic)
Last year's developments were fast enough for me to be somewhat more relaxed on this issue... (however, there is still slowing core aging rate/neuroplasticity loss down, which acts on shorter timelines, and still important if you want to do your best work)
https://twitter.com/search?q=from%3A%40RokoMijic%20immortality&src=typed_query
I don't know whether to believe, but it's a reasonable take...
Two days ago I wrote a tweet with similar idea and it get unexpected boost - which means that it is important thing for people: 'Perhaps, we don't need superintelligent AI to solve aging quickly. Some more mundane level of AI will be enough." https://twitter.com/turchin/status/1719315694813130896
EY wrote against the statement, saying that why not all these people not signing for cryonics
M. Batin wrote in favor of this statement and even made a counter-appearance during AI Pause meeting in SF https://twitter.com/MikhailBatin/status/1719948531233140754
I also agree with the statement. I'm guessing most people who haven't been sold on longtermism would too.
When people say things like "even a 1% chance of existential risk is unacceptable", they are clearly valuing the long term future of humanity a lot more than they are valuing the individual people alive right now (assuming that the 99% in that scenario above is AGI going well & bringing huge benefits).
Related question: You can push a button that will, with probability P, cure aging and make all current humans immortal. But with probability 1-P, all humans die. How high does P have to be before you push? I suspect that answers to this question are highly correlated with AI caution/accelerationsim
1-4
If I choose P=1, then 1-P=0, so I am immortal and nobody dies
2
Well sure, but the interesting question is the minimum value of P at which you'd still push
1
I think the point of the statement is: wait until the probability of you dying before you next get an opportunity to push the button is > 1-P then push the button.
You have a very strange understanding of politics, wherein you have laymen who want to advance their interests at the expense of other peoples, who realize that would be unpopular if they stated exactly what they were doing, and then as a consequence lie on Twitter about needing to do it for a different reason. This is insufficiently cynical. People almost always genuinely think their political platform is the Right and Just one. It's just that people also have strong instincts to align themselves with whatever tribe is saying they're the Good Guys, and th...
1
You are right, there are three possible avenues of approaching this: (1) people have certain goals and lie about them to advance their interests, (2) people have certain goals, and they self-delude about their true content so that they advance their interests, (3) people don't have any goals, they are simply executing certain heuristics that proved to be useful in-distribution (Reward is not an optimisation target approach), I omitted the last one from the post. But think that my observation about (2) having non-zero chance of explaining variance in opinions still stands true. And this is even more true for people engaged in AI safety, such as members of Pause AI, e/acc and (to a lesser extent) academics doing research on AI.
Even if (3) has more explanatory power, it doesn't really defat the central point of the post, which is the ought question (which is a bit of a evasive answer, I admit).
My limited impression of "e/accs", and you may think this is unfair, is that most of them seem not to have any gears-level model of the problem at all, and have instead claimed the mantle because they decided amongst each other it's the attire of futurism and libertarianism. George Hotz will show up to the Eliezer/Leahy debates with a giant American flag in the background and blurt out stuff like "Somalia is my preferred country", not because he's actually going to live there, but because he thinks that sounds based and the point of the discussion for him is to wave a jersey in the air. I don't think Hotz has made the expected value calculation you mention because I don't think he's even really gotten to the point of developing an inside view in the first place.
4
In other words, they are based-tuned stochastic parrots? Seems harsh, but the Hotz-Yudkowsky 'debate' can only be explained by something in the vicinity of this hypothesis AFAICT (haven't seen others).
Based on personal experience, you are definitely not the only one thinking about that Statement.
The statement seems like it's assuming:
-
we know roughly how to build AGI
-
we decide when to do that
-
we use the time between now and then to increase chance of successful alignment
-
if we succeed in alignment early enough, you and your loved ones won't die
I don't think any of these are necessarily true, and I think the ways they are false is asymmetric in a manner that favors caution
2
It's also assuming:
1. We know roughly how to achieve immortality
2. We can do that exactly in the window of "the last possible moment" of AGI.
3. Efforts between immortality and AGI are fungible and exclusive, or at least related in some way.
4. Ok, yeah - we have to succeed on BOTH alignment and immortality to keep any of us from dying.
3 and 4 are, I think, the point of the post. To the extent that work on immortality rather than alignment, we narrow the window of #2, and risk getting neither.
4
Isn't the assumption that once we successfully align AGI, it can do the work on immortality? So "we" don't need to know how beyond that.
I think most humans agree with this statement in an "I emotionally want this" sort of way. The want has been sublimated via religion or other "immortality projects" (see The Denial of Death). The question is, why is it taboo, and it is taboo in the sense you say? (a signal of low status)
I think these elements are at play most in peoples mind, from lay people to rationalists:
- It's too weird to think about: Considering the possibility of a strange AI-powered world where either complete extinction or immortality are possible feels "unreal". Our instinct that e
I have expressed versions of this Statement, or at least parts of it, to those close to me. They are not in the EA or LW communities and so I have to rephrase a lot. Mostly along the lines of, "One way or another, the last generation to die of old age has most likely already been born. I'm not quite sure which one, but if we manage to survive the century as a species, then I expect it's my own, and that my nieces and nephew will be able to live for millennia. I hope I'm wrong about which generation it is."
For me, both AI and medicine are far outside my areas of expertise, so I'm focusing on other problems that I have a better chance of influencing, hence the passive framing. This is even more true for my family members.
Off-topic, but somewhat related. I want to know if there is anyone reading these words who is willing to admit that he or she is kind of hoping humanity will go extinct because humanity has been unfair to him or her or because (for some other reason) humanity is bad or unworthy.
Honestly, I haven’t seen much about individual biological immortality, or even significant life-extension, in the last few years.
I suspect progress on computational consciousness-like mechanisms has fully eclipsed the idea that biological brains in the current iteration are the way of the future. And there’s been roughly no progress on upload, so the topic of immortality for currently-existing humans has mostly fallen away.
Also, if/when AI is vastly more effective than biological intelligence, it takes a lot of the ego-drive away for the losers.
Who said anything about slowly and painfully?? I don't think a fully reflective you would endorse that.
(FWIW I'm not whoever downvoted your comments, though these definitely aren't to my taste and could be interpreted as threatening)
1
Me. And I don't actually endorse that. That was my point.
And my comments are threatening? Saying I'd defend with my life my loved ones from those (like you) who are happy for them to die to achieve their goals? Sure, I guess, but I stand by that.
I'm very much of an egoist but I don't really agree with this statement. Even if the last smidge of my altruism were deleted, I still wouldn't rob banks or run a fraudulent asset exchange because I have a very cushy position in society as it is and cooperating with humanity in fact makes my life much better than not. I similarly wouldn't do existentially crazy things to try to reach the singularity for me personally. Even if normal people wouldn't understand how I'm burning the commons, you guys would.
I'm like way more likely to do norm-breaking stuff (lik...
I think a lot of people haven't fully considered how amazing immortality is. Your 7th observation is probably very accurate too. Socially sophisticated people want others to believe they're fighting for the greater good rather than their own self interest. That doesn't mean they're necessarily lying.
I fully agree with the Statement but also support Pause/Stop. The reason is that I'm highly confident that everyone dies if we achieve AGI in the near future, so since I'm still relatively young I prefer getting to live some more years instead.
Not much to add content wise but just to be one data point: I strongly agree and have talked about it with exactly one person. I have no plans to advance the position openly and proactively, largely for reasons stated, but would quietly and strongly support efforts to that end should I become aware of them.
As I've put it to my friend: the worst day for a transcendent AI to emerge is the week after I would die and the best day is the week before. We then argue over where "tomorrow" fits on that spectrum
I'm very glad that you're raising this topic for discussion. I've recently been thinking about the same thing myself. If I could delay the advent of AGI/TAI by, say, 100 years, would I? There are at least two relevant considerations here: 1) the commonly held (but not, to me, obviously true) assumption that delaying AGI/TAI increases the probability of it being safely deployed and 2) the costs of delaying AGI/TAI.
(2) motivates thinking really hard about whether (1) is true. General concern about AI safety also motivates thinking hard about whether (1) is true, since if delaying AGI/TAI does not increase the probability of a safe deployment, then we should think about what would increase it.
You should consider cross-posting to EAF. Would be interesting to see the poll differences, but might be fiddly to set up.
If I take the statement face value as written, s-risks are very relevant to disagreement - a life lived longer without solving sources of suffering tends to result in accruing damage, and saving my life immediately before I would die is one of the more likely ways to force me to live with the maximum possible damage dealt by (what just nearly killed me, and cumulative health troubles in general).
I notice that "don't die" is rarely a terminal value. Most people who say they want immortality are making some assumptions about the quality of life it would entail -- both in an s-risk sense and in a "technically people who are permanently comatose on life support aren't actually dead" sense as well.
I wouldn't call "don't die" a terminal value in the cryonics case either, because in that case you do die for awhile but then you get better. Eventual reincarnation after a delay seems meaningfully distinct from immortality, because immortality implies continuity of life.
Statement: I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.
This Statement confuses me. There are several observations I can make about it. There are also many questions I want to ask but have no idea how to answer. The goal of this post is to deconfuse myself, and to get feedback on the points that I raised (or failed to raise) below.
First observation: The Statement is directly relevant to LW interests.
It ties together the issues of immortality and AI risk, both of which are topics people here are interested in. There are countless threads, posts and discussions about high-level approaches to AI safety, both in the context of "is" (predictions) and "ought" (policy). At the same time, there is still a strong emphasis on the individual action, deliberating on which choices to make to improve the to marginal effects of living life in a certain way. The same is true for immortality. It has been discussed to death, both from the high-level and from the individual, how-do-I-sign-up-for-Alcor point of view. The Statement has been approached from the "is", but not from the "ought" perspective. At the same time:
Second observation: No one talks about the Statement.
I have never met anyone who expressed this opinion, neither in-person nor online, even after being a part (although, somewhat on the periphery) of the rationalist community for several years. Not only that, I have not been able to find any post or comment thread on LW or SSC/ACX that discusses it, argues for or against it, or really gives it any attention whatsoever. I am confused by this since the Statement seems to be fairly straightforward.
One reason might be the:
Third observation: Believing in the Statement is low status, as it constitutes an almost-taboo opinion.
Not only no one is discussing it, but the few times when I expressed the Statement in person (at EA-infiltrated rationalists meetups), it was treated with suspicion or hostility. Although to be honest, I'm not sure how much this is me potentially misinterpreting the reactions. I got the impression that it is seen as sociopathic. Maybe it is?
Fourth observation: Believing in the Statement is incompatible with long-termism, and it runs counter to significantly valuing future civilisation in general.
Fifth observation: Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.
Most people value their lives, and the lives of those around them to a much greater extent than those far away from them. This is even more true for the future lives. The revealed-preference discount factor is bounded away from 1.
Sixth observation: The Statement is internally consistent.
I don't see any problems with it on the purely logical level. Rational egoism (or variants thereof) constitutes a valid ethical theory, although it is potentially prone to self-defeat.
Seventh observation: Because openly admitting to believing in the Statement is disadvantageous, it is possible that many people in fact hold this opinion secretly.
I have no idea how plausible this is. Judging this point is one of my main goals in writing this post. The comments are a good place for debating the meta-level points, but, if I am right about the cost of holding this opinion - not so much for counting its supporters. An alternative is this anonymous poll I created - please vote if you're reading this.
Eighth observation: The Statement has the potential to explain some of the variance of attitudes to AI risk-taking.
One way of interpreting this observation might be that people arguing against pausing/for accelerating AI developments are intentionally hiding their motivations. But, using the standard Elephant-in-the-brain arguments about people's tendency to self-delude about their true reasons for acting or believing in certain things, it seems possible that some of the differences between Pause-aligned and e/acc-aligned crowd come down to this issue. Again, I have no idea how plausible this is. (People in e/acc are - usually - already anonymous, which is a point against my theory).
Ninth observation: The Statement is not self-defeating in the sense of failure of ethical theories to induce coordination in the society/escaping prisoner dilemmas.
Although the Statement relies on the ethical judgement of gambling the entire future of our civilisation on my meagre preferences for my life and life of people around me, everyone's choices are to a greater or lesser extent aligned (depending on the similarity of life expectancy). Most probably we, as the "current population" survive together or die together - with everyone dying being the status quo. Trading off future lives does not create coordination problems (ignoring the possibility of acausal trade), leaving only the conflict between different age groups.
This leads me to the...
Tenth observation: Openly resolving the topic of the Statement leads to a weird political situation.
As we said, the Statement implicitly refers to the Pareto frontier of AI existential risk-taking, which is parametrised by the life expectancy of the individual. Therefore, assuming a sufficiently powerful political system, it is a negotiation about the cut-off point - who gets left behind and who gets to live indefinitely. There does not seem to be much negotiation space - fairly compensating the first group seems impossible. Thus, this seems like a high-conflict situation. In a real-life not-so-powerful system, it might lead to unilateral action of people left below the cut-off, but they might not have enough resources to carry out the research and development alone. Also, the probabilistic nature of that decision would probably not trigger any strong outrage response.
Eleventh observation: Much of the "working EA theory" might break when applied to the Statement.
How much do people posture altruism, and how much do they really care? EA signalling can be a good trade (10% of your income in exchange for access to the network and being well-liked overall). This was the argument SSC was making in one of the posts: the beauty of the EA today is that the inside doesn't really matter - being motivated by true altruism and being motivated by status are both still bringing the same result at the end. This can, however, break if the stakes are raised sufficiently high. (I personally approach the problem of death with a pascal-wager-like attitude, but even with a discount rate of 0.9, the choice comes down to a 10x life value).
Twelfth observation: The Statement does not trivialise either way: the optimal amount of neither recklessness nor caution is zero.
The latter is a consensus (at least on LW). The former is not. The only take I ever heard advocating for this was that cryonics + certainly aligned AI is worth more in expectation than the chance of aligned AI in our lifetimes. For obvious reasons, I consider this to be a very weak argument.
Thirteenth observation: For a median believer of the Statement, we might be passing the point where the balance is being too heavily weighted towards pause.
The fundamental difference between the pause side and the accelerate side is that pause has a much bigger social momentum to it. Governments are good at delaying and introducing new regulations - not so much at speeding up and removing them (at least not recently). I can't find a source, but I think EY admitted this in one of the interviews - the only good thing about the government is that it will grind everything to a complete stop. After seeing literally all major governments take interest (US executive order, UK summit), the academia heavily leaning towards pause, and the general public being more and more scared of AI - I worry that the pause is going to be too entrenched, will have too much political momentum to stop and reverse course. I don't know how much should I consider the market forces being a counterweight.
Fourteenth observation: Taking into account s-risks in addition to x-risk might change the tradeoff.
A final argument about which I am unsure - it seems plausible that the Statement should not be endorsed by anyone because of the potential of unaligned AI to cause suffering. But I do not think s-risks are sufficiently probable to pause indefinitely.
So: Am I the only one thinking those thoughts? Am I missing any obvious insights? Is that really a taboo view? Was it debated before and resolved definitely one way or another? Is this some standard philosophical problem that already has a Google-able name? Is there any downside to openly discussing this whole thing that I don't see?