Can we talk about changing the world? Or saving the world?
I think few here would give an estimate higher than 95% for the probability that humanity will survive the next 100 years; plenty might put a figure less than 50% on it. So if you place any non-negligible value on future generations whose existence is threatened, reducing existential risk has to be the best possible contribution to humanity you are in a position to make. Given that existential risk is also one of the major themes of Overcoming Bias and of Eliezer's work, it's striking that we don't talk about it more here.
One reason of course was the bar until yesterday on talking about artificial general intelligence; another factor are the many who state in terms that they are not concerned about their contribution to humanity. But I think a third is that many of the things we might do to address existential risk, or other issues of concern to all humanity, get us into politics, and we've all had too much of a certain kind of argument about politics online that gets into a stale rehashing of talking points and point scoring.
If we here can't do better than that, then this whole rationality discussion we've been having comes to no more than how we can best get out of bed in the morning, solve a puzzle set by a powerful superintelligence in the afternoon, and get laid in the evening. How can we use what we discuss here to be able to talk about politics without spiralling down the plughole?
I think it will help in several ways that we are a largely community of materialists and expected utility consequentialists. For a start, we are freed from the concept of "deserving" that dogs political arguments on inequality, on human rights, on criminal sentencing and so many other issues; while I can imagine a consequentialism that valued the "deserving" more than the "undeserving", I don't get the impression that's a popular position among materialists because of the Phineas Gage problem. We need not ask whether the rich deserve their wealth, or who is ultimately to blame for a thing; every question must come down only to what decision will maximize utility.
For example, framed this way inequality of wealth is not justice or injustice. The consequentialist defence of the market recognises that because of the diminishing marginal utility of wealth, today's unequal distribution of wealth has a cost in utility compared to the same wealth divided equally, a cost that we could in principle measure given a wealth/utility curve, and goes on to argue that the total extra output resulting from this inequality more than pays for it.
However, I'm more confident of the need to talk about this question than I am of my own answers. There's very little we can do about existential risk that doesn't have to do with changing the decisions made by public servants, businesses, and/or large numbers of people, and all of these activities get us straight into the world of politics, as well as the world of going out and changing minds. There has to be a way for rationalists to talk about it and actually make a difference. Before we start to talk about specific ideas to do with what one does in order to change or save the world, what traps can we defuse in advance?
I myself would be disappointed if over half of LW put the probability of a single biological human (not an upload, not a reconstruction - an actual descendent with the appropriate number of ancestors alive today) alive in 100 years under 95%. I would consider that to be a gross instance of all kinds of biases. I'm not going to argue about scenarios, here, just point out that there any scenario which tends inevitably to wipe out humanity within one lifetime is totally unimaginable. That doesn't mean implausible, but it does mean improbable.
Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events ("something that's never happened before which causes X" where X is a defined consequence such as humanity's extinction) to within even an order of magnitude for p/(1-p). I doubt anybody can get even to within two orders of magnitude consistently. (I also doubt that this hypothesis of mine will be clearly decidable within the next 20 years, so I'm not particularly inclined to listen to philosophical arguments from people who'd like to discard it.)
What I'm saying is, we should stop trying to put numbers on this without big error bars. And I've yet to see anybody propose an intelligent way to deal with probabilities like 10^(-6 +/- 4); just meta-averaging it over the distribution of possible probabilities, to come up with something like 10^-3 seems to be discarding data and to lead to problems. However, that's the kind of probability I'd put on this lemma. ("Earth made uninhabitable by normal cosmic event and rescue plans fail" would probably put a floor somewhere above 10^-22 per year.)
"The chance we're all wrong about something totally unprecedented has got to be less than 99.9%" is total hubris. Yes, totally unprecedented things happen every day. But telling yourselves stories about AGI and foom does not make these stories likely.
This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences. But if you're all getting that carried away, then Less Wrong should just be renamed More Wrong.
Oh, also, I'd accept that the risk of humanity being seriously hosed within 100 years, or extinct within 1000 years, is significant - say, 10^(-3 +/- 4) which meta-averages to something like 15%.
("Seriously hosed" means gigadeath events, total enslavement, or the like. Note that we're already moderately hosed and always have been, but that seriously hosed is still distinguishable.)