There's a fairly simple and common conflict that I find explains a lot of procrastination:
I want X to have been done.
I do not want to be doing X.
So you struggle to convince yourself to "voluntarily" do X even though it's unpleasant, or you don't do X and struggle with worrying about the consequences of not doing X. :/
Does any process in which they ended up the way they did without considering your decision procedure count as #2? Like, suppose almost all the other agents it expects to encounter are CDT agents that do give in to extortion, and it thinks the risk of nuclear war with the occasional rock or UDT agent is worth it.
Especially in the age of AGI, leaders may no longer need to respect the values of most people because they're not economically relevant.
Or militarily relevant. Traditionally, if you were a ruler, you had to at least keep your army happy. However, if you command an entirely automated army that doesn't have any actual people in it, there's no risk of the army turning against you. You have the robot weapons and nobody else does, so you can do whatever the hell you want to people without having to care what anyone else thinks.
Most pivotal acts I can easily think of that can be accomplished without magic ASI help amount to "massively hurt human civilization so that it won't be able to build large data centers for a long time to come." I don't know if that's a failure of imagination, though. (An alternative might be some kind of way to demonstrate that AI existential risk is real in a way that's as convincing as the use of nuclear weapons at the end of World War II was for making people consider nuclear war an existential risk, so the world gets at least as paranoid about AI as it is about things like genetic engineering of human germlines. I don't actually know how to do that, though.)
Honestly? By going to the list of doctors that my health insurance will pay for, or some other method of semi-randomly choosing among licensed professionals that I hope doesn't anti-correlate with the quality of their advice. There are probably better ways, but I don't know what they are offhand. ::shrug::
If you were accused of a crime and intended to plead not guilty, how would you choose a defense attorney, assuming you weren't going to use a public defender?
And of course the right answer is “absolutely everyone”. It should be fully public. If your setup is such that it even makes sense to ask this question of “who should be allowed to know what cryptographic algorithm we use”, then your security system is a complete failure and nobody should trust you with so much as their mother’s award-winning recipe for potato salad, much less any truly sensitive data.
This makes sense for computer security, but for biosecurity it doesn't work, because it's a lot harder to ship a patch to people's bodies than to people's computers. The biggest reason there has never been a terrorist attack with a pandemic-capable virus is that, with few exceptions (such as smallpox), we don't know what they are.
A: My understanding is that the U.S. Government is currently funding research programs to identify new potential pandemic-level viruses.
K: Unfortunately, yes. The U.S. government thinks we need to learn about these viruses so we can build defenses — in this case vaccines and antivirals. Of course, vaccines are what have gotten us out of COVID, more or less. Certainly they’ve saved a ton of lives. And antivirals like Paxlovid are helping. So people naturally think, that’s that’s the answer, right?
But it’s not. In the first place, learning whether a virus is pandemic capable does not help you develop a vaccine against it in any way, nor does it help create antivirals. Second, knowing about a pandemic-capable virus in advance doesn’t speed up research in vaccines or antivirals. You can’t run a clinical trial in humans on a new virus of unknown lethality, especially one which has never infected a human — and might never. And given that we can design vaccines in one day, you don’t save much time in knowing what the threat is in advance.
The problem is there are around three to four pandemics per century that cause a million or more deaths, just judging from the last ones — 1889, 1918, 1957, 1968 and 2019. There’s probably at least 100 times as many pandemic-capable viruses in nature — it’s just that most of them never get exposed to humans, and if they do, they don’t infect another human soon enough to spread. They just get extinguished.
What that means is if you identify one pandemic-capable virus, even if you can perfectly prevent it from spilling over and there’s zero risk of accidents, you’ve prevented 1/100 of a pandemic. But if there’s a 1% chance per year that someone will assemble that virus and release it, then you’ve caused one full pandemic in expectation. In other words, you’ve just killed more than 100 times as many people as you saved.
you can’t very well trust someone else more than you trust yourself
In certain domains, I absolutely can and will do this, because "someone else" has knowledge and experience that I don't and could not conveniently acquire. For example, if I hire lawyers for my business's legal department, I'm probably not going to second-guess them about whether a given contract is unfair or contains hidden gotchas, and I'm usually going to trust a doctor's diagnosis more than I trust my own. (The shortfalls of "Doctor Google" are well-known, so although I often do "do my own research" I only trust it so much.)
I think the amount of cash a bank loses in a typical armed robbery really isn't that large compared to the amounts of money the bank actually handles - bank robbers are a nusiance but not an existential threat to the bank.
The actual big danger to banks comes from insiders; as the saying goes, the best way to rob a bank is to own one.
I actually have found an example of a strategy that doesn't incentivize someone else to self-modify into Hawkbot: https://www.lesswrong.com/posts/TXbFFYpNWDmEmHevp/how-to-give-in-to-threats-without-incentivizing-them
Basically, when you're faced with a probable extorter, you play Cooperate some of the time (so you don't always get nuked) but either Defect or Nuke back often enough that Hawkbot gets a lower expected value than Cooperate/Cooperate.