I do not argue that my idea is sane; however I think your critique doesn't do it justice. So let me briefly point out that:
measuring probabilities of world destruction is very hard; being able to measure them at the 1e-12 level seems very, very hard
It's enough to use upper bounds. If we have e.g. an additional module to check our AI source code for errors, and such a module decreases probability of one of the bits being flipped, we can use our risk budget to calculate how many modules at minimum we need. Etc.
How should it decide what budget level to give itself?
It doesn't. You don't build any intelligent system without a risk budget. Initial budgets are distributed to humans, e.g. 10^-15 to each human alive in 2016.
looks like simple utility maximization (go to the movie if the benefits outweigh the costs) gives the right answer
If utility is dominated by survival of humanity, then simple utility maximization is exactly the same as reducing total "existential risk emissions" in the sense I want to use them above.
Whether or not your utility is dominated by survival of humanity is an individual question.
the budget replenishes
Not at all. A risk budget is decreased by your best estimate of your total risk "emission", which is what fraction of the future multiverse (weighted by probability) you spoiled.
So I think budgets are the wrong way to think about this--they rely too heavily on subjective perceptions of risk, they encourage being too cautious (or too risky) instead of seeing tail risks as linear in probability, and they don't update on survival when they should.
Quite likely they are - but probably not for these reasons.

Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
The page http://lesswrong.com/r/discussion/new/ returns error for me for 12 hours, but other pages are fine. Is it only my glitch?
error text: "You have encountered an error in the code that runs Less Wrong. The site maintainers have been informed and will get to it is as soon as they can. In the unlikely event that you've bumped into this error before and think that no-one is paying attention, please report the error and how to reproduce it on http://code.google.com/p/lesswrong/issues/list'
If the error is localised you might still find awesome Less Wrong content in the Main article area or in the Discussion area.
There was an issue with how the linkposts handled unicode URLs, fixed by wezm here and here.