Very surprised there's no mention here of Hanson's "Foom Liability" proposal: https://www.overcomingbias.com/p/foom-liability

I think that apocalypse insurance isn't as satisfactory as you imply, and I'd like to explain why below.

First: what's a hardline libertarian? I'll say that a hardline libertarian is in favour of people doing stuff with markets, and there being courts that enforce laws that say you can't harm people in some small, pre-defined, clear-cut set of ways. So in this world you're not allowed to punch people but you are allowed to dress in ways other people don't like.

Why would you be a hardline libertarian? If you're me, the answer is that (a) markets and freedom etc are pretty good, (b) you need ground rules to make them good, and (c) government power tends to creep and expand in ill-advised ways, which is why you've got to somehow rein it in to only do a small set of clearly good things.

If you're a hardline libertarian for these reasons, you're kind of unsatisfied with this proposal, because it's sort of subjective - you're punishing people not because they've caused harm, but because you think they're going to cause harm. So how do you assess the damages? Without further details, it sounds like this is going to involve giving a bunch of discretion to a lawmaker to determine how to punish people - discretion that could easily be abused to punish a variety of activities that should thrive in a free society.

There's probably some version that works, if you have a way of figuring out which activities cause how much expected harm that's legibly rational in a way that's broadly agreeable. But that seems pretty far-off and hard. And in the interim, applying some hack that you think works doesn't seem very libertarian.

I'm confused, like I always confused with hardline libertarianism. Why would companies agree to this? Who would put capabilities researchers in jail if they, like, "I'd rather not purchase apocalypse insurance and create AI anyway"? Why this actor is not a state by other name? What should I read to became less confused?

[ note: I am not a libertarian, and haven't been for many years. But I am sympathetic. ]

Like many libertarian ideas, this mixes "ought" and "can" in ways that are a bit hard to follow.  It's pretty well-understood that all rights, including the right to redress of harm, are enforced by violence.  In smaller groups, it's usually social violence and shared beliefs about status.  In larger groups, it's a mix of that, and multi-layered resolution procedures, with violence only when things go very wrong.  

When you say you'd "prefer a world cognizant enough of the risk to be telling AI companies that...", I'm not sure what that means in practice - the world isn't cognizant and can't tell anyone anything.  Are you saying you wish these ideas were popular enough that citizens forced governments to do something?  Or that you wish AI companies would voluntarily do this without being told?  Or something else?

In theory, one billionth of the present buys one billionth of the future: Go to a casino, put all on black until you can buy the planet.

Therefore, they can buy their insurance policy with dollars. If you can't buy half the planet, apparently you can't afford a 50% chance to kill everyone.

Here's another angle that a hardline libertarian might take (partly referenced in the footnote).  I'm not quite sure how far I'd want to take the underlying principle, but let's run with it for now:

Libertarians generally agree that it's permissible to use (proportionate) force to stop someone from engaging in aggressive violence.  That could mean grabbing someone's arm if they're punching in your direction, even if they haven't hit you yet.  You don't necessarily have to wait until they've harmed you before you use any force against them.  The standard for this is probably something like "if a reasonable person in your position would conclude with high probability that the other person is trying to harm you".

Next, let's imagine that you discover that your neighbor Bob is planning to poison you.  Maybe you overhear him telling someone else about his plans; and you notice a mail delivery of a strange container labeled "strychnine" to his house, and look up what it means; maybe you somehow get hold of his diary, which describes his hatred for you and has notes about your schedule and musings about how and when would be the best time to add the poison to your food.  At some point, a reasonable person would have very high certainty that Bob really is planning to kill you.  And then you would be justified in using some amount of force, at least to stop him and possibly to punish him.  For tactical reasons it's best to bring in third parties, show them the evidence, and get them on your side; ideally this would all be established in some kind of court.  But in principle, you would have the right to use force yourself.

Or, like, suppose Bob is setting up sticks of dynamite right beside your house.  Still on his property!  But on the edge, as close to your house as possible.  And setting up a fuse, and building a little barrier that would reduce the amount of blast that would reach his house.  Surely at some point in this process you have the right to intervene forcibly, before Bob lights the fuse.  (Ideally it'd be resolved through speech, but suppose Bob just insists that he's setting it up because it would look really cool, and refuses to stop.  "Does it have to be real dynamite?"  "Yeah, otherwise it wouldn't look authentic."  "I flat out don't believe you.  Stop it."  "No, this is my property and I have the right.")

Next, suppose Bob is trying to set up a homemade nuclear reactor, or perhaps to breed antibiotic-resistant bacteria, for purposes of scientific study.  Let's say that is truly his motive—he isn't trying to endanger anyone's lives.  But he also has a much higher confidence in his own ability to avoid any accidents, and a much higher risk tolerance, than you do.  I think the principle of self-defense may also extend to here: if a reasonable person believes with high confidence that Bob is trying to do a thing that, while not intended to harm you, has a high probability of causing serious harm to you, then you have the right to use force to stop it.  (Again, for tactical reasons it's best to bring in third parties, and to try words before actually using force.)

If one is legitimately setting up something like a nuclear reactor, ideally the thing to do would be to tell everyone "I'm going to set up this potentially-dangerous thing.  Here are the safety precautions I'm going to take.  If you have objections, please state them now."  [For a nuclear reactor, the most obvious precaution is, of course, building it far away from humans, but I think there are exceptions.]  And probably post signs at the site with a link to your plans.  And if your precautions are actually good, then in theory you reach a position where a reasonable person would not believe your actions have a high chance of seriously harming them.

The notion of "what a reasonable person would believe" is potentially very flexible, depending on how society is.  Which is dangerous, from the perspective of "this might possibly justify a wide range of actions".  But it can be convenient when designing a society.  For example, if there's a particular group of people who make a practice of evaluating dangerous activities, and "everyone knows" that they're competent, hard-nosed, have condemned some seriously bad projects while approving some others that have since functioned successfully... then you might be at the point where you could say that an educated reasonable person, upon discovering the nuclear reactor, would check with that group and learn that they've approved it [with today's technology, that would mean the reactor would have a sign saying "This project is monitored and approved by the Hard-Nosers, approval ID 1389342", and you could go to the Hard-Nosers' website and confirm this] before attempting to use force to stop it; and an uneducated reasonable person would at least speak with a worker before taking action, and learn of the existence of the Hard-Nosers, and wouldn't act before doing some research.  In other words, you might be able to justify some kind of regulatory board being relevant in practice.

(There might be layers to this.  "People running a legitimate nuclear reactor would broadcast to the world the fact that they're doing it and the safety precautions they're taking!  Therefore, if you won't tell me, that's strong evidence that your precautions are insufficient, and justifies my using force to stop you!"  "Granted.  Here's our documentation."  "You also have to let me inspect the operation—otherwise that's evidence you have something to hide!"  "Well, no.  This equipment is highly valuable, so not letting random civilians come near it whenever they want is merely evidence that we don't want it to be stolen."  "Ok, have as many armed guards follow me around as you like."  "That's expensive."  "Sounds like there's no way to distinguish your operation from an unsafe one, then, in which case a reasonable person's priors would classify you as an unacceptable risk."  "We do allow a group of up to 20 members of the public to tour the facility on the first of every month; join the next one if you like."  "... Fine."

And then there could be things where a reasonable person would be satisfied with that; but then someone tells everyone, "Hey, regularly scheduled inspections give them plenty of time to hide any bad stuff they're doing, so they provide little evidentiary value", and then if those who run the reactor refuse to allow any unscheduled inspections, that might put them on the wrong side of the threshold of "expected risk in the eyes of a reasonable external observer".  So what you'd effectively be required to do would change based on what that random person had said in public and to you.  Which is not, generally, a good property for a legal system.  But... I dunno, maybe the outcome is ok?  This would only apply to people running potentially dangerous projects, and it makes some kind of sense that they'd keep being beholden to public opinion.)

So we could treat increasingly-powerful AI similarly to nuclear reactors: can be done for good reasons, but also has a substantial probability of causing terrible fallout.  In principle, if someone is building a super-AI without taking enough precautions, you could judge that they're planning to take actions that with high-enough probability are going to harm you badly enough that it would be proper self-defense for you to stop them by force.

But, once again, tactical considerations make it worth going to third parties and making your case to them, and it's almost certainly not worth acting unless they're on your side.  Unfortunately, there is much less general agreement about the dangers of AI (and the correct safety strategies) than about the dangers of nuclear reactors, and it's unlikely in the near future that you'd get public opinion / the authorities on your side about existing AI efforts (though things can change quickly).  But if someone did take some unilateral (say maybe 1% of the public supported them) highly destructive action to stop a particular AI lab, that would probably be counterproductive: they'd go to jail, it would discredit anyone associated with them, martyr their opposition, at best delay that lab by a bit, and motivate all AI labs to increase physical security (and possibly hide what they're doing).

For the moment, persuading people is the only real way to pursue this angle.

The answer is that apocalypse insurance—unlike liability insurance—must pay out in advance of the destruction of everyone. If somebody wishes to risk killing you (with some probability), there's presumably some amount of money they could pay you now, in exchange for the ability to take that risk.

Pretty sure you mean they should pay premiums rather than payouts? 

I like the spirit of this idea, but think it's both theoretically and practically impossible: how do you value apocalypse? Payouts are incalculable/infinite/meaningless if no one is around. 

The underlying idea seems sound to me: there are unpredictable civilizational outcomes resulting from pursuing this technology -- some spectacular, some horrendous -- and the pursuers should not reap all the upside when they're highly unlikely to bear any meaningful downside risks. 

I suspect this line of thinking could be grating to many self-described libertarians who lean e/acc and underweight the possibility that technological progress != prosperity in all cases. 

It also seems highly impractical because there is not much precedent for insuring against novel transformative events for which there's no empirical basis*. Good luck getting OAI, FB, MSFT, etc. to consent to such premiums, much less getting politicians to coalesce around a forced insurance scheme that will inevitably be denounced as stymying progress and innovation with no tangible harms to point to (until it's too late).

Far more likely (imo) are post hoc reaction scenarios where either:

a) We get spectacular takeoff driven by one/few AI labs that eat all human jobs and accrue all profits, and society deems these payoffs unfair and arrives at a redistribution scheme that seems satisfactory (to the extent "society" or existing political structures have sufficient power to enforce such a scheme)

b) We get a horrendous outcome and everyone's SOL

* Haven't researched this and would be delighted to hear discordant examples.

+4. This doesn't offer a functional proposal, but it makes some important points about the situation and offers an interesting reframe, and I hope it gets built upon. Key paragraph:

In other words: from a libertarian perspective, it makes really quite a lot of sense (without compromising your libertarian ideals even one iota) to look at the AI developers and say "fucking stop (you are taking far too much risk with everyone else's lives; this is a form of theft until and unless you can pay all the people whose lives you're risking, enough to offset the risk)".

I like this proposal. It's a fun rethinking of the problem.

However,

1. How can you even approximate a fair price for these payouts? AI risks are extremely conditional and depend on difficult to quantify assumptions. "The model leaked AND optimized itself to work on computers insecure and Internet available at the time of escape AND humans failed to stop it AND..."

For something like a nuclear power plant for instance, most of the risk is all black swans. There are a ton of safety systems and mechanisms to cool the core. We know from actual accidents that when this fails, it's not because each piece of equipment all failed at the same time. This relates to AI risk because multiplying the series probability does not tell you the true risk.

For all meltdowns i am aware of, the risk happened because human operators or an unexpected common cause made all the levels of safety fail at once.

1. 3 mile island : operators misunderstood the situation and turned off cooling.
2. Chernobyl: operators bypassed the automated control system with patch cables and put the core into an unstable part of the operating curve.
3. Fukushima : plant wide power failure, road conditions prevented bringing spare generators on site quickly.

Each cause is coupled, and can thought of as a single cause. Adding n+1 serial defenses might not have helped in each case (depends what it is)

If AI does successfully kill everyone it's going to be in a way humans didn't model.

2. Mineshaft gap argument. Large fees on AI companies simply encourages them to set up shop in countries that don't charge the fees. In futures where they don't kill everyone, those countries will flourish or will conquer the planet. So the other countries have to drop these costs and subsidize hasty catch up ASI research or risk losing. In the futures where AI does attack and try to kill everyone, not have tool AI (aligned only with the user) increases the probability that the AI wins. Most defensive measures are stronger if you have your own AI to scale production. (More bunkers, more nukes to fire back, more spacesuits to stop the bio and nano attacks, more drones...)

I am sorry to say that on a forum where many people are likely to have been raised in a socio-cultural environnement where libertarian ideas are deeply rooted. My voice will sound dissonant here and I call to your open-mindedness.

I think that there are strong limitations to such ideas as developed in the OP proposal. Insurance is mutualization of risk, it's a statistic approach relying on the possibility to assess a risk. It works for risks happening frequently, with a clear typology, like car accidents, tempest, etc. Even in these cases there is always an insurance ceiling. But risks that are exceptionnal and the most hazardous, like war damages, nuclear accident etc, cannot be insured and are systematically subject to contractual exclusions. There is no apocalypse insurance because the risk cannot be assessed by actuaries. Even if you create such an insurance, it would be artificial, non rationally assessed, with an insurance ceiling making it useless. There is even the risk that it gives the illusion that everything is ok and acceptable. The insurance mechanism does not encourages responsability, but a contrario irresponsability. On top of that compensation through money is a legal fiction. But in real life money isn't everything that's worth. In the most dramatic cases the real damage is never repaired (i.e. loss of your child, loss of your legs, loss of your own life), it's more a symbolic compensation, "better than nothing".

As a matter of fact, I have professionnal knowledge of law and insurance, from inside, and I have a very practical experience of what I am saying. Libertarianism encourages an approach that is very theoretical and economics-centered, and that's honestly interesting, but it is also somehow disconnected from reality. Just one ordinary example among others. A negligent fourniture mover destroyed family goods inherited from generations, not a word of excuses because he said "there are insurances for that". In the end, after many months of procedure and inenumerable time and energy spent by the victim, the professional's insurance paid almost nothing because of course old family goods have no economical value for experts. Well, when you see how insurance effectively works in real cases, and how it can often encourages negligent and irresponsible behavior, it is very difficult to be enthousiast at the idea that AI existential hazard could be managed by the subscription of an insurance policy.

Great post. I agree with almost all of this. What I am uncertain about is the idea that AI existential risk is a rights violation under the most strict understanding of libertarianism. 

As another commenter has suggested, we can't claim that any externality creates rights to stop or punish a given behavior, or libertarianism turns into safetyism.^[1] If we take the Non-Aggression Principle as a common standard for a hardline libertarian view of what harms give you a right to restitution or retaliation, it seems that x-risk does not fit this definition. 

1.The most clear evidence seems to be that Murray Rothbard wrote the following:

“lt is important to insist [...] that the threat of aggression be palpable, immediate, and direct; in short, that it be embodied in the initiation of an overt act. [...] Once we bring in "threats" to person and property that are vague and future--i.e., are not overt and immediate--then all manner of tyranny becomes excusable.” (The Ethics of Liberty p. 78) 

X-risk by its very nature falls into the category of "vague and future."

 2. To take your specific example of flying planes over someone's house, a follower of Rothbard, Walter Block, has argued that this exact risk is not a violation of the non-aggression principle. He also states that risks from nuclear power are "legitimate under libertarian law." (p. 295)^[2] If we consider AI analogous to these two risks, it would seem Block would not agree that there is a right to seek compensation for x-risk.

3. Matt Zwolinski criticized the NAP for having an "all-or-nothing attitude toward risk" as it does not indicate what level of risk constitutes aggression. Another libertarian writer responded that a risk that constitutes a direct "threat" is aggression, (i.e. pointing a pistol at someone, even if this doesn't result in the victim being shot) but risks of accidental damage are not aggression unlese these risks are imposed with threats of violence: 

"If you don’t wish to assume the risk of driving, then don’t drive. And if you don’t want to run the risk of an airplane crashing into your house, then move to a safer location. (You don’t own the airspace used by planes, after all.)" 

This implies to me that Zwolinski's criticism is accurate with regards to accidents, which would rule out x-risk as a NAP violation.
 

Conclusion

This shows that at least some libertarians' understanding of rights does not include x-risk as a violation. I consider this to be a point against their theory of rights, not an argument against pursuing AI safety. The most basic moral instinct suggests that creating a significant risk of destroying all of humanity and its light-cone is a violation of the rights of each member of humanity.^[3] 

While I think that not including AI x-risk (and other risks/accidental harms) in its definition of proscribable harms means that the NAP is too narrow, the question still stands as to where to draw the line as to what externalities or risks give victims a right to payment, and which do not. I'm curious where you draw the line. 

It is possible that I am misunderstanding something about libertarianism or x-risk that contradicts the interpretation I have drawn here.

Anyway, thanks for articulating this proposal. 
 

1. ^{^}

   See also this argument by Alexander Volokh:

   "Some people’s happiness depends on whether they live in a drug-​free world, how income is distributed, or whether the Grand Canyon is developed. Given such moral or ideological tastes, any human activity can generate externalities […] Free expression, for instance, will inevitably offend some, but such offense generally does not justify regulation in the libertarian framework for any of several reasons: because there exists a natural right of free expression, because offense cannot be accurately measured and is easy to falsify, because private bargaining may be more effective inasmuch as such regulation may make government dangerously powerful, and because such regulation may improperly encourage future feelings of offense among citizens."

2. ^{^}

   Block argues that it would be wrong for individuals to own nuclear weapons, but he does not make clear why this is a meaningful distinction.

3. ^{^}

    And any extraterrestrials in our light-cone, if they have rights. But that's a whole other post. 

I agree with most of this, but as a hardline libertarian take on AI risk it is incomplete since it addresses only how to slow down AI capabilities. Another thing you may want a government to do is speed up alignment, for example through government funding of R&D for hopefully safer whole brain emulation. Having arbitration firms, private security companies, and so on enforce proof of insurance (with prediction markets and whichever other economic tools seem appropriate to determine how to set that up) answers how to slow down AI capabilities but doesn’t answer how to fund alignment.

One libertarian take on how to speed up alignment is that

(1) speeding up alignment / WBE is a regular public good / positive externality problem (I don’t personally see how you do value learning in a non-brute-force way without doing much of the work that is required for WBE anyway, so I just assume that “funding alignment” means “funding WBE”; this is a problem that can be solved with enough funding; if you don’t think alignment can be solved by raising enough money, no matter how much money and what the money can be spent on, then the rest of this isn’t applicable)

(2) there are a bunch of ways in which markets fund public goods (for example, many information goods are funded by bundling ads with them) and coordination problems involving positive or negative externalities or other market failures (all of which, if they can in principle be solved by a government by implementing some kind of legislation, can be seen as / converted into public goods problems, if nothing else the public goods problem of funding the operations of a firm that enforces exactly whatever such a legislation would say; so the only kind of market failure that truly needs to be addressed is public goods problems)

(3) ultimately, if none of the ways in which markets fund public goods works, it should always still be possible to fall back on Coasean bargaining or some variant on dominant assurance contracts, if transaction costs can be made low enough

(4) transaction costs in free markets will be lower due, among other reasons, to not having horridly inefficient state-run financial and court systems

(5) prediction markets and dominant assurance contracts and other fun economic technologies don’t, in free markets, have the status of being vaguely shady and perhaps illegal that they have in societies with states

(6) if transaction costs cannot be made low enough for the problem to be solved using free markets, it will not be solved using free markets

(7) in that case, it won’t, either, be solved by a government that makes decisions through, directly or indirectly, some kind of voting system, because for voters to vote for good governments that do good things like funding WBE R&D instead of bad things like funding wars is also an underfunded public good with positive externalities and the coordination problem faced by voters involves transaction costs that are just as great as those faced by potential contributors to a dominant assurance contract (or to a bundle of dominant assurance contracts), since the number of parties, amount of research and communication needed, and so on are just as great and usually greater, and this remains true no matter the kind of voting system used, whether that involves futarchy or range voting or quadratic voting or other attempts at solving relatively minor problems with voting; so using a democratic government to solve a public goods or externality problem is effectively just replacing a public goods or externality problem by another that is harder or equally hard to solve.

In other words: from a libertarian perspective, it makes really quite a lot of sense (without compromising your libertarian ideals even one iota) to look at the AI developers and say "fucking stop (you are taking far too much risk with everyone else's lives; this is a form of theft until and unless you can pay all the people whose lives you're risking, enough to offset the risk)".

Yes, it makes a lot of sense to say that, but not a lot of sense for a democratic government to be making that assessment and enforcing it (not that democratic governments that currently exist have any interest in doing that). Which I think is why you see some libertarians criticize calls for government-enforced AI slowdowns.