This is a linkpost for https://x.com/gcolbourn/status/2121198431938957578
New Comment
Pause AI US is obviously bad for the world, so it would be bad to fund. Control AI is less obviously bad for the world, but yes, it also is very bad for the world according to many people who look into it (and as far as I can tell is causing on-net less advocacy to happen because of how they go about doing things). You can disagree, but it's silly to walk around saying that everything that looks on the tin good to you hence is immediately evidence of catastrophic judgement if someone doesn't support it.
PauseIA France got their primary funding ask filled. They could have received a bit more money, but marginal money to them seemed less valuable than marginal funding to e.g. Encode, who are knocking it out of the park and seem in a better position to scale up.
By my estimate, only ~11% of Lightcone Commons funding [...] are relevant for the short-timelines ASI-alignment-intractable world we are most likely in.
PauseAI Global ($1.497M); PauseAI Deutschland ($38k); CeSIA ($198k); AI Safety Camp, which has a “Stop/Pause AI” track ($255k); The Future Society ($469k); Centre for Long-Term Resilience (CLTR) ($1.082M); Oxford China Policy Lab ($1.601M); Legal Advocates for Safe Science and Technology (LASST) ($1.012M). Total = $6.152M, out of total funding of $63.13M => 10%.
This is just a silly list. Really, you think The AI Future Project is not doing work relevant to navigating short timelines alignment-intractable worlds? Daniel and the AIFP team has been maybe the primary group building a case for short timelines and the need for some kind of immediate slowdown. Then you also exclude Encode, who have been among the primary drivers of both lab advocacy and state regulation with SAIP. Somehow you even exclude Palisade Research who seem most responsible for getting Bernie Sanders to call for an immediate international moratorium on AI.
This just doesn't really make any sense. You can argue about marginal cost-effectiveness and that Lightcone Commons evaluators should focus more on funding things neglected by other funders, but trying to argue that most of its grants aren't relevant to short timelines worlds with not remotely enough time to solve alignment seems just clearly wrong.
4
1PauseAI US is not obviously bad for the world. I think Holly has actually moved the Overton Window in a positive direction re it being morally unacceptable to work at a frontier AI company, and Anthropic actually being (very) bad.
Control AI is less obviously bad for the world, but yes, it also is very bad for the world according to many people who look into it (and as far as I can tell is causing on-net less advocacy to happen because of how they go about doing things)
This is badly in need of a citation, or multiple citations. How the hell it it causing less advocacy to happen?
you think The AI Future Project is not doing work relevant to navigating short timelines alignment-intractable worlds
They've done enough! As have many of the other funded orgs. We don't need more research, on the margin. We need more laws passed! I'm not saying these orgs shouldn't exist. But they shouldn't be getting funded at the expense of other, more useful, interventions not getting funded.
you also exclude Encode
No, they were included (see footnote 6).
Palisade Research who seem most responsible for getting Bernie Sanders to call for an immediate international moratorium on AI.
Explain? My understanding is that Control AI and Evitable (David Krueger) did a lot there (and MIRI). The only mentions of Sanders on Palisade's website are here and here (and there's no mention of them engaging with him).
1Imagine you have an abusive boss in your organization working on your "advocacy" department. Should you fire that boss, or increase their budget? Obviously bad operators in a space can cause harm. People can make working in a space more painful or costly. People can lie and steal and cheat and all kinds of complicated adversarial stuff. Clearly you wouldn't say "How the hell could Anthropic cause less AI Safety to happen?" or "How the hell could cG funding dynamics cause less good work to happen?". Things often harm the thing that they say they are trying to support on the tin. Such is life. Coordination and doing good things is hard.
I think Greg asked for specific mechanisms by which you think ControlAI is net negative, whereas you replied with an explanation that people genuinely caring about X can end up being very counterproductive wrt X.
1ControlAI, a nonprofit devoted to banning superintelligence, said it’s been working closely with Casar and Sanders’s teams on the bill.
If you are looking for a specific example of ControlAI being misleading, I have talked to two ex-employees with ControlAI both of whom have a bunch of context on the Sander's bill and both considered this statement highly misleading. I and they have seen no evidence that ControlAI was in any meaningful way "working closely" with Casar and Sandar's teams on the bill.
1I don't know about Eliezer and Nate, but I'm certainly donating based on a worldview that is fairly different from yours.
We appear to be on track for some sort of anti-AI US law before ASI. We're much less on track for that law to be well thought out. Palisade is more likely than the orgs you support to help politicians figure out which laws would target the main risks.
11
1
Lightcone Commons have completed their first granting round. And the things their granters and recommenders have funded mostly do not meet the current moment (the same is true for SFF’s most recent round, and Coefficient Giving). Let’s hope that their upcoming future rounds do better, or other funders step in. All of our lives are on the line.
(Apologies if this is blunt, but this is important..)
I get that money allocators (grantmakers and VCs alike) typically invest in people over ideas. But this is taking that to the extreme. Yudkowsky and Soares’ funding recommendations don’t fit with their book (which I am a big fan of)! They are funding obscure content creators, researchers and think tanks that are mostly just people in their Bay Area social circles, it seems. Why aren’t they funding orgs directly aimed at getting international agreement to ban RSI and ASI, as per what they say is necessary (and is, in fact, necessary), in chapter 14 of IABIED? I heard that orgs such as Control AI were in their funding orders, but why so low as to not have any real chance of being seriously funded?[1] Soares talks about organisations he’d recommend funding here, but is seemingly unaware that they already exist[2]. The other evaluators’ choices don’t seem any better (but at least some of them have excuses). $813k for Mox, $300k for Vibecamp, and only $75k for PauseIA [France] and $50k for PauseAI UK? (None for PauseAI Global or PauseAI US?) Seriously?
Other good grants: Stop The AI Race ($400k[3]); CeSIA ($765k); Secure AI project ($858k); American Shield ($1M). And on the margin, it’s good that campaign group Encode AI was given $2M, but they don’t even mention extinction risk on their website![4]
We need to be funding orgs that are consistent in warning of extinction risk, and how close we are to it. Most of the funded orgs don’t. Indeed, most mainstream media outlets now mention extinction more than AI Safety org websites!
Ultimately, only law can prevent extinction[5], so we need to be focused on funding orgs that are directly aimed at getting relevant laws passed, with the ultimate goal of global extinction-preventing law - i.e. international agreements banning RSI and ASI.
Timelines are likely very short now. We probably only have a few months left to sort things out before the point of no return (extinction becoming inevitable). The only way we survive now is via political solutions. There is just no time for technical research to pan out (if it’s even possible). We need to focus on creating more advocacy. And we need to be on the exploit end of the explore/exploit tradeoff - i.e. scaling up already successful initiatives, not funding ever more new ones. Rationality and EA are meant to be about being objective and empirical, and avoiding cognitive biases and tribalism. Instead there seems to be a lot of social and personal bias on display here.
By my estimate, only ~11% of Lightcone Commons funding[6], and only ~10% of SFF’s most recent funding round[7], (and ~4% of CG’s “Navigating Transformative AI” grants over the last 6 months[8] [9]) are relevant[10] for the short-timelines ASI-alignment-intractable world we are most likely in. The rest of the funding has very little chance of being relevant or having a non-negligible impact (i.e. of reducing the chances of our extinction) in time[11].
Lightcone Commons could be doing so much better, and I hope they do with their upcoming rapidfire grant rounds[12]. And so could SFF. And Coefficient Giving. But in the meantime, it looks like we need to redouble efforts to find major funders outside the LW/EA ecosystem, who are willing to seriously support political efforts, if we are actually going to survive.
Thanks to Gabe Alfour for helpful suggestions and feedback (any shortcomings are mine).
Or is it mainly just down to Habryka’s petty grudge against a couple of the founders? Tragic if true.
I guess he might argue that these existing orgs aren’t exactly what he has in mind, but given short timelines, I think scaling existing orgs (possibly with (financial) nudges to change their strategy slightly) is much more likely to be productive than starting new orgs from scratch.
? It’s hard to read the data, as presented.
Nathan Calvin seems to be doing good, relevant work, at least.
It’s quite unfortunate that the guy who wrote the linked post’s funding recommendations mostly don’t match up to his words here!
PauseIA: $75k; PauseAI UK: $50k; Stop The AI Race: $400k; Secure AI project: $858k; CeSIA: $765k; American Shield: $1M; Encode AI: $2M. Total = $5.148M, out of total funding of $45.1M, => 11%.
PauseAI Global ($1.497M); PauseAI Deutschland ($38k); CeSIA ($198k); AI Safety Camp, which has a “Stop/Pause AI” track ($255k); The Future Society ($469k); Centre for Long-Term Resilience (CLTR) ($1.082M); Oxford China Policy Lab ($1.601M); Legal Advocates for Safe Science and Technology (LASST) ($1.012M). Total = $6.152M, out of total funding of $63.13M => 10%.
Simon Institute for Longterm Governance: $5.206M); Good Ancestors New Zealand: $750k. Total = $5.956M out of a total of $158.8M => 4%.
Some factors that might be relevant for CG (and also SFF): Anthropic shareholding by the main funders; (mistaken) belief that a “defence in depth” strategy will be able to contain ASI.
The boundary for what is relevant is somewhat fuzzy, hence the use of “~”.
To be clear: many are worthwhile projects, and it’s better that they are being done (and funded) than not, but they are unlikely to actually help us stay alive.
If anyone at Lightcone is taking this feedback on board, I’ve applied to be an evaluator. I’ve been focused on AI Notkilleveryonism for the last 3.5 years (AI Safety in general for much longer), and have donated over $600k of my own money towards this, including $100k to Control AI, $105k to PauseAI Global and $90k to PauseAI US.