But where did that number come from? At some point, an intelligent system that was not handed a budget selects a budget for itself. Presumably the number is set according to some cost-benefit criterion, instead of chosen because it's three hands worth of fingers in a log scale based on two hands worth of fingers.

Of course, my point is to build all intelligent systems so that they do not hand themselves a new budget, with probability that is within our risk budget (which we choose arbitrarily).

If it isn't, how do you expect the agent to actually stick to such a budget?

I hope that survival of humanity dominates the utility function of people who build AI, and they will do their best to carry it over to the AI. You can individually have another utility function, if it serves you well in your life. (As long as you won't build any AIs). But that was a wrong way to answer your previous point:

One, it looks like simple utility maximization (go to the movie if the benefits outweigh the costs) gives the right answer, and being more or less cautious than that suggests is a mistake (at least, of how the utility is measured).

Not in case of multiple agents, who cannot easily coordinate. E.g. what if each human's utility function makes it look reasonable to have a 1/1000 risk of destroying the world, for potential huge personal gains?

If I view the seven pulls as independent events, it depletes my budget by 7/6, but if I treat them as one event, it depletes my budget by only 1-(5/6)^7, which is about 72%.

I am well aware of this, but the effect is negligible if we speak of small probabilities.

It is my impression that there are at least some examples in which this is done in practice: as far as I know, in rocket design you do in fact calculate those for most components, including software used on the on-board computers. This information is used to e.g. decide on the amount of duplication of electronics components in critical systems of the rocket. I am, however, not an expert on rockets.

It seems plausible that at least in some concepts, we can indeed build safeguards that have a certain efficiency that we know at reducing our overall risk. Even if this is true only sometimes, than it would be useful to have a way to calculate the maximum allowed risk levels for extinction-like events.

Incidentally, I am also of the opinion that having any kind of calculation would work better than making a non-zero extinction risk taboo, or not subject to negotiation (which seems to be the case currently).

However of course, I am not claiming that my idea is so great. I stand behind my opinion that we need some such system to make sensible tradeoffs on "emissions" of existential risk.

The upper bound is nearly always that there a black swan reason that makes you destroy the world.

Ah, I see you added this part.

I generally agree. Still, sometimes you'll want something to guide your design even if you know that there might be some such black swan. You are surely not suggesting that existence of black swans is enough to make us abandon all effort and do whatever.

Sorry, that was unclear. I meant "internal" to the decision making process. This process is implemented on many individuals in case of human collective intelligence. But in some cases it makes sense to think about it as a single decision making process in the abstract.

Yes. as I mentioned in other comments, in practice you build safeguards and they give you some reduction of the upper bound on risk. So you use your risk budget to calculate how many safeguards you need to build.

I do not argue that my idea is sane; however I think your critique doesn't do it justice. So let me briefly point out that:

measuring probabilities of world destruction is very hard; being able to measure them at the 1e-12 level seems very, very hard

It's enough to use upper bounds. If we have e.g. an additional module to check our AI source code for errors, and such a module decreases probability of one of the bits being flipped, we can use our risk budget to calculate how many modules at minimum we need. Etc.

How should it decide what budget level to give itself?

It doesn't. You don't build any intelligent system without a risk budget. Initial budgets are distributed to humans, e.g. 10^-15 to each human alive in 2016.

looks like simple utility maximization (go to the movie if the benefits outweigh the costs) gives the right answer

If utility is dominated by survival of humanity, then simple utility maximization is exactly the same as reducing total "existential risk emissions" in the sense I want to use them above.

Whether or not your utility is dominated by survival of humanity is an individual question.

the budget replenishes

Not at all. A risk budget is decreased by your best estimate of your total risk "emission", which is what fraction of the future multiverse (weighted by probability) you spoiled.

So I think budgets are the wrong way to think about this--they rely too heavily on subjective perceptions of risk, they encourage being too cautious (or too risky) instead of seeing tail risks as linear in probability, and they don't update on survival when they should.

Quite likely they are - but probably not for these reasons.

Yes, but to implement risk budgets it's enough to know upper bounds with reasonable certainty. It is possible to implement verifiable upper bounds, esp. in tech contexts such as AI.

If you think this is related, then I failed to communicate my idea.

You can think of risk contracts as of an internal currency of decision making, that allows to coordinate risk management in a situation when there are multiple agents acting independently (or new agents are created).

Definitely different from betting on extinction events, or betting on predictions about those events.

But there's a strong negative connection with "special care" and "special needs"...

Using your money analogy, what I'm saying above is that if you expect the item to cost $20, you will go to the shop with only $20 or $25 in your wallet. So you won't buy the item if it costs $100.

This theory is compatible with willpower being expendable - obviously, you can't carry more cash in your wallet than the total amount you own.

So it is a more detailed model, in which you can be short of money in two ways: you can't afford this at all, or you didn't think to have that much cash on you when you left the house.