If you spend 8000 times less on AI alignment (compared to the military),
You must also believe that AI risk is 8000 times less (than military risk).
Why?
We know how to effectively spend money on the military: get more of what we have and do R&D to make better stuff. The only limit on effective military spending is all the other things that the money is needed for, i.e. having a country worth defending.
It is not clear to me how to buy AI safety. Money is useless without something to spend it on. What would you buy with your suggested level of funding?
I think that's a very important question, and I don't know the answer for what we should buy.
However, suppose not knowing what you should spend on, dramatically decreases the total amount you should spend (e.g. by 10x). If that was really true in general, then imagine a country with a large military discovers that its enemies are building very powerful drone swarm weapons, which can easily destroy all its tanks, aircraft carriers, and so forth very cheaply.
Military experts are all confused and in disagreement how to counter these drone swarms, just like the AI alignment community. Some of them say that resistance is futile, and the country is "doomed." Others have speculative ideas like using lasers. Still others say that lasers are stupid, because the enemy can simply launch the swarms in bad weather and the lasers won't reach them. Just like with AI alignment, there are no proven solutions, and every solution tested against drone swarms are destroyed pathetically.
Should the military increase its budget, or decrease its budget, since no one knows what you can spend money on to counter the drone swarms?
I think the moderate, cool headed response is to spend a similar amount, exploring all the possibilities, even without having any ideas which are proven to work.
Uncertainty means the expected risk reduction is high
If we are uncertain about the nature of the risk, we might assume that 50%, spending more money reduces the risk by a reasonable amount (similar to risks we do understand), and possibly even more due to discovering brand new solutions instead of getting marginal gains on existing solutions. And 50%, spending more money is utterly useless, because we are at the mercy of luck.
Therefore, the efficiency of spending on AI risk should be at least half the efficiency of spending on military risk, or at least within the same order of magnitude. This argument argues over orders of magnitude.
If increasing the time for alignment by pausing AI can work, so can increasing the money for alignment
Given that we effectively have a race between capabilities and alignment, the relative spending on capabilities and alignment seems important.
A 2x capabilities decrease should be similar in effect to a 2x alignment increase, or at least a 4x alignment increase.
The only case where decreasing capabilities funding works far better than increasing alignment funding, is if we decrease capabilities funding to zero, using extremely forceful worldwide regulation and surveillance. But that would also require governments to freak out about AI risk (prioritize it as highly as military risk), and benefit from this letter.
Money isn’t magic. It’s nothing more than the slack in the system of exchange. You have to start from some idea of what the work is that needs to happen. That seems to me to be lacking. Are there any other proposals on the table against doom but “shut it all down”?
Suppose you had literally no ideas at all how to counter drone swarms, and you were really bad at judging other people's ideas for countering drone swarms. In that case, would you, upon discovering that your countries adversaries developed drone swarms, (making your current tanks and ships obsolete), decide to give up on military spending, and cut military spending by 100 times?
Please say you would or explain why not.
My opinion is that you can't give up (i.e. admit there is a big problem but spend extremely little on it) until you fully understood the nature of the problem with certainty.
Money isn't magic, but it determines the number of smart people working on the problem. If I was a misaligned superintelligence, I would be pretty scared of a greater amount of human intelligence working to stop me from being born in the first place. They get only one try, but they might actually stumble across something that works.
Suppose you had literally no ideas at all how to counter drone swarms, and you were really bad at judging other people's ideas for countering drone swarms.
In that case, I would be unqualified to do anything, and I would be wondering how I got into a position where people were asking me for advice. If I couldn’t pass the buck to someone competent, I’d look for competent people, get their recommendations, try as best I could to judge them, and turn on the money tap accordingly. But I can’t wave a magic wand, and where there was a pile of money there is now a pile of anti-drone technology.
Neither can anyone in AI alignment.
If everyone else is also unqualified because the problem is so new, and every defence they experimented with got obliterated by drone swarms, then you would agree they should just give up, and admit military risk remains a big problem but spend far less on it, right?
So if no one else knew how to counter drone swarms, and every defence they experimented with got obliterated by drone swarms,
…then by hypothesis, you’re screwed. But you’re making up this scenario, and this is where you’ve brought the imaginary protagonists to. You’re denying them a solution, while insisting they should spend money on a solution.
I think just because every defence they experimented with got obliterated by drone swarms, doesn't mean they should stop trying, because they might figure out something new in the future.
It's a natural part of life to work on a problem without any idea what the solution will be like. The first people who studied biology had no clue what modern medicine would look like, but their work was still valuable.
Being unable to imagine a solution does not prove a solution doesn't exist.
Sure, never give up, die with dignity if it comes to that. None of that translates into a budget. Concrete plans translate into a budget.
At some point there has to be concrete plans, yes without concrete plans nothing can happen.
I'm probably not the best person in the world to decide how the money should be spent, but one vague possibility is this:
Probably not good enough for you :/ but I imagine someone else can come up with a better plan.
- If you spend 8000 times less on AI alignment (compared to the military),
- You must also believe that AI risk is 8000 times less (than military risk).[1]
No. You must believe that spending on military is 8000 times more helpful to your goals. And really, in a democracy or other multilateral decision framework, nobody actually has to believe this, it just has to be 8000 times easier to agree to spend a marginal amount, which is quite path-dependent.
Even if you DO believe the median estimates as given, you have to weight it by the marginal change that spending makes. Military spending keeps the status quo, rewards your constituents, makes you look good, etc. AI spending is ... really confusing and doesn't really help any political goals. It's absolutely not clear that spending more can increase safety - the obvious thing that happens when you spend is acceleration, not slowdown.
If you believe that spending more on safety leads to acceleration instead, you should try to refute my argument for why it is a net positive.
I'm honestly very curious how my opponents will reply to my "net positive" arguments, so I promise I'll appreciate a reply and upvote you.
I pasted it in this comment so you don't have to look for it:
Why I feel almost certain this open letter is a net positive
Delaying AI capabilities alone isn't enough. If you wished for AI capabilities to be delayed by 1000 years, then one way to fulfill your wish is if the Earth had formed 1000 years later, which delays all of history by the same 1000 years.
Clearly, that's not very useful. AI capabilities have to be delayed relative to something else.
That something else is either:
Progress in alignment (according to optimists like me)
or
Either way, the AI Belief-Consistency Letter speeds up that progress by many times more than it speeds up capabilities. Let me explain.
Case 1:
Case 1 assumes we have a race between alignment and capabilities. From first principles, the relative funding of alignment and capabilities matters in this case.
Increasing alignment funding by 2x ought to have a similar effect to decreasing capability funding by 2x.
Various factors may make the relationship inexact, e.g. one might argue that increasing alignment by 4x might be equivalent to decreasing capabilities by 2x, if one believes that capabilities is more dependent on funding.
But so long as one doesn't assume insane differences, the AI Belief-Consistency Letter is a net positive in Case 1.
This is because alignment funding is only at $0.1 to $0.2 billion, while capabilities funding is at $200+ billion to $600+ billion.
If the AI Belief-Consistency Letter increases both by $1 billion, that's a 5x to 10x alignment increase and only a 1.002x to 1.005x capabilities increase. That would clearly be a net positive.
Case 2:
Even if the wildest dreams of the AI pause movement succeed, and the US, China, and EU all agree to halt all capabilities above a certain threshold, the rest of the world still exists, so it only reduces capabilities funding by 10x effectively.
That would be very good, but we'll still have a race between capabilities and alignment, and Case 1 still applies. The AI Belief-Consistency Letter still increases alignment funding by far more than capabilities funding.
The only case where we should not worry about increasing alignment funding, is if capabilities funding is reduced to zero, and there's no longer a race between capabilities and alignment.
The only way to achieve that worldwide, is to "solve diplomacy," which is not going to happen, or to "go nuclear," like Eliezer Yudkowsky suggests.
If your endgame is to "go nuclear" and make severe threats to other countries despite the risk, you surely can't oppose the AI Belief-Consistency Letter on the grounds that "it speeds up capabilities because it makes governments freak out about AGI," since you actually need governments to freak out about AGI.
Conclusion
Make sure you don't oppose this idea based on short term heuristics like "the slower capabilities grow, the better," without reflecting on why you believe so. Think about what your endgame is. Is it slowing down capabilities to make time for alignment? Or is it slowing down capabilities to make time for governments to freak out and halt AI worldwide?
You make a very good point about political goals, and I have to agree that this letter probably won't convince politicians whose political motivations prevent them from supporting AI alignment spending.
Yes, military spending indeed rewards constituents, and some companies go out of their way to hire people in multiple states etc.
PS: I actually mentioned the marginal change in a footnote, but I disabled the sidebar so maybe you missed it. I'll add the sidebar footnotes back.
Thanks!
IMO it's unclear what kind of person would be influenced by this. It requires the reader to a) be amenable to arguments based on quantitative probabilistic reasoning, but also b) overlook or be unbothered by the non sequitur at the beginning of the letter. (It's obviously possible for the appropriate ratio of spending on causes A and B not to match the magnitude of the risks addressed by A and B.)
I also don't understand where the numbers come from in this sentence:
In order to believe that AI risk is 8000 times less than military risk, you must believe that an AI catastrophe (killing 1 in 10 people) is less than 0.001% likely.
Hi,
By a very high standard, all kinds of reasonable advice are non-sequitur. E.g. a CEO might explain to me "if you hire Alice instead of Bob, you must also believe Alice is better for the company than Bob, you can't just like her more," but I might think "well that's clearly a non-sequitur, just because I hire Alice instead of Bob doesn't imply Alice is better for the company than Bob. Since maybe Bob is a psychopath who would improve the company's fortunes by committing crime and getting away with it, so I hire Alice instead."
X doesn't always imply Y, but in cases where X doesn't imply Y there has to be an explanation.
In order for the reader to agree that AI risk is far higher than 1/8000th the military risk, but still insist that 1/8000th the military budget is still justified, he would need a big explanation, e.g. the marginal benefit of spending 10% more on the military reduces military risk by 10%, but the marginal benefit of spending 10% more on AI risk somehow only reduces AI risk by 0.1%, since AI risk is far more independent of countermeasures.
It's hard to have such drastic differences, because one needs to be very certain that AI risk is unsolvable. If one was uncertain of the nature of AI risk, and there existed plausible models where spending a lot reduces the risk a lot, then these plausible models dominate the expected value of risk reduction.
Thank you for pointing out that sentence, I will add a footnote for it.
If we suppose that military risk for a powerful country (like the US) is lower than the equivalent of a 8% chance of catastrophe (killing 1 in 10 people) by 2100, then 8000 times less would be a 0.001% chance of catastrophe by 2100.
I will also add a footnote for the marginal gains.
Thank you, this is a work in progress, as the version number suggests :)
I agree wholeheartedly with the sentiment. I also agree with the underlying assumptions made in the Compendium[1], that it would really require a Manhattan project level of effort to understand:
All of which is to say, that I believe these problems are resolvable, but only if, to your point, a significant amount of expenditure, and the greatest minds in this generation are set to the task of resolving them ahead of the deployment of a superintelligent system. We face a Manhattan Project level of risk, but we are not acting as if we are facing that systematically.
Dear policymakers,
We demand that the AI alignment budget be Belief-Consistent with the military budget.
Belief-Consistency is a simple yet powerful idea:
Yet the only way to reach Belief-Consistency, is to
Greatly increase AI alignment spending,
or
Let us explain:
In order to believe that AI risk is 8000 times less than military risk, you must believe that an AI catastrophe (killing 1 in 10 people) is less than 0.001% likely.[2]
This clearly contradicts the median AI expert, who sees a 5%-12% chance of an AI catastrophe (killing 1 in 10 people), the median superforecaster, who sees 2.1%, other experts, who see 5%, and the general public, who sees 5%.
In fact, assigning only 0.001% chance contradicts the median expert so profoundly, that you need to become 99.95% certain that you won't realize you were wrong and the majority of experts were right!
If there was more than a 0.05% possibility that you study the disagreement further, and realize the risk exceeds 2% just like most experts say, then the laws of probability forbid you from believing a 0.001% risk.
We believe that foreign invasion concerns have decreased over the last century, and AGI concerns have increased over the last decade, but budgets remained within the status quo, causing a massive inconsistency between belief and behaviour.
Do not let humanity's story be so heartbreaking.
Version 1.1us
These numbers are for the US, but the same argument applies to NATO and other countries.
Please add your signature by asking in a comment (or emailing me). Thank you so truly much.
Signatories:
References
Claude 3.5 drew the infographic for me.
“Military Budget: $800 billion”
“AI Safety: $0.1 billion”:
“the median AI expert, who sees a 5%-12% chance of an AI catastrophe (killing 1 in 10 people), other experts, who see 5%, the median superforecaster, who sees 2.1%, and the general public, who sees 5%.”
We should assumes the marginal risk reduction is similar for military risk and AI risk, i.e. we don't have lopsided situations where spending 10% more on the military reduces military risk by 10%, but spending 10% more on AI risk only reduces AI risk by 0.1%.
This is a very reasonable assumption to make since "you can't give up (admit there's big problem but spend extremely little) until you fully understand the nature of the problem with certainty."
If there was only a 50% chance that spending 10% more on AI risk reduces AI risk by 10%, then spending 10% more on AI risk reduces AI risk by at least 5%.
This assumes that the military risk for a powerful country (like the US) is less than the equivalent of a 8% chance of catastrophe (killing 1 in 10 people) by 2100.
USAFacts Team. (August 1, 2024). “How much does the US spend on the military?” USAFacts. https://usafacts.org/articles/how-much-does-the-us-spend-on-the-military/
Wiggers, Kyle. (October 22, 2024). “The US AI Safety Institute stands on shaky ground.” TechCrunch. https://techcrunch.com/2024/10/22/the-u-s-ai-safety-institute-stands-on-shaky-ground/
McAleese, Stephen, and NunoSempere. (July 12, 2023). “An Overview of the AI Safety Funding Situation.” LessWrong. https://www.lesswrong.com/posts/WGpFFJo2uFe5ssgEb/an-overview-of-the-ai-safety-funding-situation/?commentId=afv74rMgCbvirFdKp
Karger, Ezra, Josh Rosenberg, Zachary Jacobs, Molly Hickman, Rose Hadshar, Kayla Gamin, and P. E. Tetlock. (August 8, 2023). “Forecasting Existential Risks Evidence from a Long-Run Forecasting Tournament.” Forecasting Research Institute. p. 259. https://static1.squarespace.com/static/635693acf15a3e2a14a56a4a/t/64f0a7838ccbf43b6b5ee40c/1693493128111/XPT.pdf#page=260
Stein-Perlman, Zach, Benjamin Weinstein-Raun, and Katja Grace. (August 3, 2022). “2022 Expert Survey on Progress in AI.” AI Impacts. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
Why this open letter might succeed even if previous open letters did not
Politicians don't reject pausing AI because they are evil and want a misaligned AI to kill their own families! They reject pausing AI because their P(doom) honestly is low, and they genuinely believe that if the US pauses then China will race ahead, building even more dangerous AI.
But as low as their P(doom) is, it may still be high enough that they have to agree with this argument (maybe it's 1%).
It is very easy to believe that "we cannot let China win," or "if we don't do it someone else will," and reject pausing AI. But it may be more difficult to believe that you are 99.999% sure of no AI catastrophe, and thus 99.95% sure the majority of experts are wrong, and reject this letter.
Also remember that AI capabilities spending is 1000x greater than alignment spending.
This difference makes it far easier for me to make a quantitative argument for increasing alignment funding, than to make a quantitative argument for increasing regulation.
I am not against asking for regulation! I just think we are dropping the ball when it comes to asking for alignment funding.
PS: I feel this letter better than my previous draft, because although it is longer, the gist of it is easier to understand and memorize: "make the AI alignment budget Belief-Consistent with the military budget."
Why I feel almost certain this open letter is a net positive
Delaying AI capabilities alone isn't enough. If you wished for AI capabilities to be delayed by 1000 years, then one way to fulfill your wish is if the Earth had formed 1000 years later, which delays all of history by the same 1000 years.
Clearly, that's not very useful. AI capabilities have to be delayed relative to something else.
That something else is either:
Progress in alignment (according to optimists like me)
or
Either way, the AI Belief-Consistency Letter speeds up that progress by many times more than it speeds up capabilities. Let me explain.
Case 1:
Case 1 assumes we have a race between alignment and capabilities. From first principles, the relative funding of alignment and capabilities matters in this case.
Increasing alignment funding by 2x ought to have a similar effect to decreasing capability funding by 2x.
Various factors may make the relationship inexact, e.g. one might argue that increasing alignment by 4x might be equivalent to decreasing capabilities by 2x, if one believes that capabilities is more dependent on funding.
But so long as one doesn't assume insane differences, the AI Belief-Consistency Letter is a net positive in Case 1.
This is because alignment funding is only at $0.1 to $0.2 billion, while capabilities funding is at $200+ billion to $600+ billion.
If the AI Belief-Consistency Letter increases both by $1 billion, that's a 5x to 10x alignment increase and only a 1.002x to 1.005x capabilities increase. That would clearly be a net positive.
Case 2:
Even if the wildest dreams of the AI pause movement succeed, and the US, China, and EU all agree to halt all capabilities above a certain threshold, the rest of the world still exists, so it only reduces capabilities funding by 10x effectively.
That would be very good, but we'll still have a race between capabilities and alignment, and Case 1 still applies. The AI Belief-Consistency Letter still increases alignment funding by far more than capabilities funding.
The only case where we should not worry about increasing alignment funding, is if capabilities funding is reduced to zero, and there's no longer a race between capabilities and alignment.
The only way to achieve that worldwide, is to "solve diplomacy," which is not going to happen, or to "go nuclear," like Eliezer Yudkowsky suggests.
If your endgame is to "go nuclear" and make severe threats to other countries despite the risk, you surely can't oppose the AI Belief-Consistency Letter on the grounds that "it speeds up capabilities because it makes governments freak out about AGI," since you actually need governments to freak out about AGI.
Conclusion
Make sure you don't oppose this idea based on short term heuristics like "the slower capabilities grow, the better," without reflecting on why you believe so. Think about what your endgame is. Is it slowing down capabilities to make time for alignment? Or is it slowing down capabilities to make time for governments to freak out and halt AI worldwide?