This is a linkpost for https://foundational-research.org/s-risks-talk-eag-boston-2017/
That's a twist on a standard LW argument, see e.g. here:
Fragility of value is the thesis that losing even a small part of the rules that make up our values could lead to results that most of us would now consider as unacceptable
It seems to me that fragility of value can lead to massive suffering in many ways.
You're basically dialing that argument up to eleven. From "losing a small part could lead to unacceptable results" you are jumping to "losing any small part will lead to unimaginable hellscapes":
with a tall sharp peak (FAI) surrounded by a pit that's astronomically deeper