Wow!
Many thanks for posting that link. It's clearly the most important thing I've read on LW in a long time, I'd upvote it ten times if I could.
It seems like an s-risk outcome (even one that keeps some people happy) could be more than a million times worse than an x-risk outcome, while not being a million times more improbable, so focusing on s-risks is correct. The argument wasn't as clear to me before. Does anyone have good counterarguments? Why shouldn't we all focus on s-risk from now on?
(Unsong had a plot point where Peter Singer declared that the most important task for effective altruists was to destroy Hell. Big props to Scott for seeing it before the rest of us.)
I don't buy the "million times worse," at least not if we talk about the relevant E(s-risk moral value) / E(x-risk moral value) rather than the irrelevant E(s-risk moral value / x-risk moral value). See this post by Carl and this post by Brian. I think that responsible use of moral uncertainty will tend to push you away from this kind of fanatical view
I agree that if you are million-to-1 then you should be predominantly concerned with s-risk, I think they are somewhat improbable/intractable but not that improbable+intractable. I'd guess the probability is ~100x lower, and the available object-level interventions are perhaps 10x less effective. The particular scenarios discussed here seem unlikely to lead to optimized suffering, only "conflict" and "???" really make any sense to me. Even on the negative utilitarian view, it seems like you shouldn't care about anything other than optimized suffering.
The best object-level intervention I can think of is reducing our civilization's expected vulnerability to extortion, which seems poorly-leveraged relative to alignment because it is much less time-sensitive (unless we fail at alignment and so end up committi...
large amounts of human utility (more than just pleasure) seem harder to achieve than large amounts of human disutility (for which pain is enough).
Carl gave a reason that future creatures, including potentially very human-like minds, might diverge from current humans in a way that makes hedonium much more efficient. If you assigned significant probability to that kind of scenario, it would quickly undermine your million-to-one ratio. Brian's post briefly explains why you shouldn't argue "If there is a 50% chance that x-risks are 2 million times worse, than they are a million times worse in expectation." (I'd guess that there is a good chance, say > 25%, that good stuff can be as efficient as bad stuff.)
I would further say: existing creatures often prefer to keep living even given the possibility of extreme pain. This can be easily explained by an evolutionary story, which suffering-focused utilitarians tend to view as a debunking explanation: given that animals would prefer keep living regardless of the actual balance of pleasure and pain, we shouldn't infer anything from that preference. But our strong dispreference for intense suffering has a similar evolutionary origin, and is no more reflective of underlying moral facts than is our strong preference for survival.
I feel a weird disconnect on reading comments like this. I thought s-risks were a part of conventional wisdom on here all along. (We even had an infamous scandal that concerned one class of such risks!) Scott didn't "see it before the rest of us" -- he was drawing on an existing, and by now classical, memeplex.
It's like when some people spoke as if nobody had ever thought of AI risk until Bostrom wrote Superintelligence -- even though that book just summarized what people (not least of whom Bostrom himself) had already been saying for years.
I guess I didn't think about it carefully before. I assumed that s-risks were much less likely than x-risks (true) so it's okay not to worry about them (false). The mistake was that logical leap.
In terms of utility, the landscape of possible human-built superintelligences might look like a big flat plain (paperclippers and other things that kill everyone without fuss), with a tall sharp peak (FAI) surrounded by a pit that's astronomically deeper (many almost-FAIs and other designs that sound natural to humans). The pit needs to be compared to the peak, not the plain. If the pit is more likely, I'd rather have the plain.
Was it obvious to you all along?
Interesting to see another future philosophy.
I think my own rough future philosophy is making sure that the future has an increase in autonomy for humanity. I think it transforms into S-risk reduction assuming that autonomous people will chose to reduce their suffering and their potential future suffering if they can. It also transforms the tricky philosophical question of defining suffering into the tricky philosophical question of defining autonomy, that might be trade that is preferred.
I think I prefer the autonomy increase because I do not have to try...
So I don't have much experience with philosophy; this is mainly a collection of my thoughts as I read through.
1) S-risks seem to basically describe hellscapes, situations of unimaginable suffering. Is that about right?
2) Two assumptions here seem to be valuing future sentience and the additive nature of utility/suffering. Are these typical stances to be taking? Should there be some sort of discounting happening here?
3) I'm pretty sure I'm strawmanning here, but I can't but feel like there's some sort of argument by definition here where we first defined s-...
We could also add a-risks: that human civilisation will destroy alien life and alien civilizations. For example, LHC-false vacuum-catastrophe or UFAI could dangerously affect all visible universe and kill an unknown number of the alien civilisations or prevent their existence.
Preventing risks to alien life is one of the main efforts in the sterilisation of Mars rovers and sinking of Galileo and Cassini in Jupiter and Saturn after the end of their missions.
Feedback: I had to scroll a very long way until I found out what "s-risk" even was. By then I had lost interest, mainly because generalizing from fiction is not useful.
If it makes sense to continue adding letters to different risks, l-risks could be identified, that is the risks that kill all life on earth. The main difference for us, humans, that there are zero chances of the new civilisation of Earth in that case.
But y-risks term is free. What could it be?
I guess I didn't think about it carefully before. I assumed that s-risks were much less likely than x-risks (true) so it's okay not to worry about them (false). The mistake was that logical leap.
In terms of utility, the landscape of possible human-built superintelligences might look like a big flat plain (paperclippers and other things that kill everyone without fuss), with a tall sharp peak (FAI) surrounded by a pit that's astronomically deeper (many almost-FAIs and other designs that sound natural to humans). The pit needs to be compared to the peak, not the plain. If the pit is more likely, I'd rather have the plain.
Was it obvious to you all along?
Didn't you realize this yourself back in 2012?