Obviously, nobody knows.
Well, do the markets price these things, then? Let me draw a difference here.
The financial markets price credit risk. This means that from observable market prices I can calculate the implied probability of default for a given entity. The financial markets price, say, the future Fed decisions on rates -- from observable market prices I can calculate the implied probability of the Fed raising or lowering the rates by a given amount at the next meeting.
I cannot calculate the implied probability of a nuclear exchange or of a pandemic from observable market prices.
With respect to your points,
I agree with (1), though I have doubts about the "better" part.
(2) seems meaningless to me: life creates bad incentives and bad behaviour. The fact that my neighbour has a shiny foozle and I don't creates incentives for me to bonk him over the head and take his foozle.
For (3) I don't know why you think so. There are financial markets in insurance and reinsurance risks (concerned with catastrophes like hurricanes and earthquakes), there are financial markets in sovereign debt (highly concerned with political risk) -- what's so special about long-term catastrophic risk?
I disagree with (4) -- I see no reason to think it's "unreasonable" :-) You're making a naked assertion, it needs supporting arguments.
(5) is an empty phrase -- everything has consequences.
So, you can calculate the implied probability of credit default because you can model credit obligations as priced by two factors only: yield and default risk. This is a crude model and won't actually predict all bond prices in the real world (though it'll come close), because people buy bonds for other reasons and the actual price of bonds tells you one fact: what people will pay for them. Everything else is just a model to get you there.
Apologies, because you know this. I'm not trying to patronize you. But please don't patronize me. The fact that much mo...
Betting on the future is a good way to reveal true beliefs.
As one example of such a bet on a key debate about a post-human future, I'd like to announce here that Robin Hanson and I have made the following agreement. (See also Robin's post at Overcoming Bias):
It's a bet on the old question: ems vs. de novo AGI. Kurzweil and Kapor bet on another well-known debate: Will machines pass the Turing Test. It would be interesting to list some other key debates that we could bet on.
But it's hard to make a bet when settling the bet may occur in extreme conditions:
MIRI has a "techno-volatile" world-view: We're not just optimistic or pessimistic about the impact of technology on our future. Instead, we predict that technology will have an extreme impact, good or bad, on the future of humanity. In these extreme futures, the fundamental components of a bet--the bettors and the payment currency--may be missing or altered beyond recognition.
So, how can we calibrate our probability estimates about extreme events? One way is by betting on how people will bet in the future when they are closer to the events, on the assumption that they'll know better than we do. Though this is an indirect and imperfect method, it might be the best we have for calibrating our beliefs about extreme futures.
For example, Robin Hanson has suggested a market on tickets to a survival shelter as a way of betting on an apocalypse. However, this only relevant for futures where shelters can help; and where there is time to get to one while the ticket holder is alive, and while the social norm of honoring tickets still applies.
We could also define bets on the progress of MIRI and similar organizations. Looking back on the years since 2005, when I started tracking this, I would have liked to bet on, or at least discuss, certain milestones before they happened. They served as (albeit weak) arguments from authority or from social proof for the validity of MIRI's ideas. Some examples of milestones that have already been reached:
Looking to the future, we can bet on some other FAI milestones. For example, we could bet on these coming true by a certain year.
(Some of these will need more refinement before we can bet on them.)
Another approach is to bet on technology trends: brain scanning resolution; prices for computing power; etc. But these bets are about a Kurzweillian Law of Accelerating Returns, which may be quite distinct from the Intelligence Explosion and other extreme futures we are interested in.
Many bets only make sense if you believe that a soft takeoff is likely. If you believe that, you could bet on AI events while still allowing the bettors a few years to enjoy their winnings.
You can make a bet on hard vs. soft takeoff simply by setting your discount rate. If you're 20 years old and think that the economy as we know it will end instantly in, for example, 2040, then you won't save for your retirement. (See my article at H+Magazine.) But such decisions don't pin down your beliefs very precisely: Most people who don't save for their retirement are simply being improvident. Not saving makes sense if the human race is about to go extinct, but also if we are going to enter an extreme utopia or dystopia where your savings have no meaning. Likewise, most people save for retirement simply out of old-fashioned prudence, but you might build up your wealth in order to enjoy it pre-Singularity, or in order to take it with you to a post-Singularity world in which "old money" is still valuable.
I'd like to get your opinion: What are the best bets we can use for calibrating our beliefs about the extreme events we are interested in? Can you suggest some more of these indirect markers, or a different way of betting?