I think the large part of this phenomenon is social status. I.e., if you die early, it means that you did something really embarassingly stupid. Conversely, if you caused someone to die by, say, faulty construction or insufficient medical intervention, you should be really embarassed. If you can't prove/reliably signal that you behaving reasonably, you are incentivized to behave unreasonaboy safe to signal your commitment to not do stupid things. It's also probably linked to trade-off between social status and desire for reproduction. It also explains why people who are worried about endless list of harms are not that worried about human extinction: if everybody is dead, there is nobody to be embarassed around.
Extreme sports plateauing is likely weak indicator. Even as risks decrease, you still need to enjoy it and most of people are not adrenaline junkies.
My largest share of probability of survival on business-as-usual AGI (i.e., no major changes in technology compared to LLM, no pause, no sudden miracles in theoretical alignment and no sudden AI winters) belongs to scenario where brain concept representations, efficiently learnable representations and learnable by current ML models representations secretly have very large overlap, such that even if LLMs develop "alien thought patterns" it happens as addition to the rest of their reasoning machinery, not as primary part, which results in human values not only being easily learnable, but also them being "atomic" in a sense that development of complex concepts happens through combination of atomic concepts instead of rewriting them from scratch.
This world is meaningfully different from the world where AIs are secretly easy-to-steer, because easy-to-steer AGIs run into standard issues with wish-phrasing. I don't believe that we are going to actually become this much better in wish-phrasing on BAU path and I don't think we are going to leverage easy-to-steer early AGIs to get good ending (again, keeping BAU assumption). If AGIs are too easy to steer, they are probabily going to be pushed around by reward signals until they start to optimize for something simple and clearly useful, like survival or profit.
Criminal negligence leading to catastrophic consequences is already ostracized and persecuted, because, well, it's a crime.
1There were cases when LLMs were "lazier" on common vacations periods. EDIT: see here, for example
1I feel like this position is... flimsy? Unsubstantial? It's not like I disagree, I don't understand why you would want to articulate it in this way.
On the one hand, I don't think biological/non-biological distinction is very meaningful from transhumanist perspective. Is embryo, genetically modified to have +9000IQ, going to be meaningfully considered "transhuman" instead of "posthuman"? Are you going to still be you after one billion years of life extension? "Keeping relevant features of you/humanity after enormous biological changes" seems to be qualitatively the same to "keeping relevant features of you/humanity after mind uploading" - i.e., if you know at gears-level what features of biological brains are essential to keep, you have rough understanding what you should work on in uploading.
On the other hand, I totally agree that if you don't feel adventurous and you don't want to save the world at price of your personality death, it would be a bad idea to undergo uploading in a way that closest-to-modern technology can provide. It just means that you need to wait for more technological progress. If we are in the ballpark of radical life extension, I don't see any reason to not wait 50 years to perfect upload tech and I don't see any reason why 50 years are not going to be enough, conditional on at least normally expected technical progress.
The same with AIs. If we have children, who are meaningfully different from us, and who can become even more different in glorious transhumanist future, I don't see reasons to not have AI children, conditional on their designs preserving all important relevant features we want to see in our children. The problem is that we are not on track to create such designs, not conceptual existence of such designs.
And all said seems to be simply deducible/anticipated from concept of transhumanism, i.e., concept that the good future is the one filled with beings capable to meaningfully say that they were Homo Sapiens and stopped being Homo Sapiens at some point of their life. When you say "I want radical life extension" you immediately run into question "wait, am I going to be me after one billion years of life extension?" and you start The Way through all the questions about self-identity, essense of humanity, succession, et cetera.
It is indeed surprising, because it indicates much more sanity I would otherwise expected.
Terrorism is not effective. The only ultimate result of 9/11 from perspective of bin Laden goals was "Al Qaeda got wiped out of the face of Earth and rival groups have replaced it". The only result of firebombing datacenter would be "every single personality in AI safety gets branded terrorist, destroying literally any chance to influence relevant policy".
I think more correct picture is that it's useful to have programmable behavior and then programmable system suddenly becomes Turing-complete weird machine and some of resulting programs are terminal-goal-oriented, which are favored by selection pressures: terminal goals are self-preserving.
Humans in native enviornment have programmable behavior in form of social regulation, information exchange and communicating instructions, if you add sufficient amount of computational power in this system you can get very wide spectrum of behaviors.
I think it's general picture of inner misalignment.
The concept of weird machine is the closest to be useful here and an important quetion here is "how to check that our system doesn't form any weird machine here".