(A response to @mabramov post from a couple days ago: https://www.lesswrong.com/posts/WCutvyr9rr3cpF6hx/forecasting-is-way-overrated-and-we-should-stop-funding-it )

TL;DR: I agree with Marcus that broadly, additional forecasting funding to the tune of tens (!?) of millions of dollars at this point would probably not yield a great return. However, I think the benefits of some forecasting funding have been tremendous. Whatever funding it initially took to help get platforms like Metaculus and Manifold off the ground has had incredible ROI. Tens of thousands of people use those sites to get information every day, even though far less are actively forecasting on them, and they provide the infrastructure by which people can ask and get crowdsourced forecasting on consequential and mundane subjects for free.

~~~~~

As I understand Marcus's argument, his central thesis is that we haven't seen the benefits of this past forecasting funding, but I think the opposite is true! Here are just a few examples:

• It's hard to measure the value of "epistemic infrastructure," not just for forecasting sites but also things like Wikipedia and OurWorldInData. That doesn't mean that value isn't there. Has Wikipedia been a good return on investment? Obviously! Manifold is far less impactful than Wikipedia, but Wikipedia gets about $200 million per year between returns on its endowment and donations. The return on investment in Manifold is probably still way higher than Marcus seems to believe. Hundreds of thousands of people have made incrementally better decisions; hundreds of thousands of people have learned to think about the world a little more concretely and quantitatively. I'm one active user of thousands on Manifold and I'd personally value its impact on my life quite highly, as I'd wager Marcus might too.
• Giant companies like Kalshi and Polymarket have grown in part because of research around how to best leverage crowdsourced forecasting. Inasmuch as they themselves have been funded, which Marcus claimed but I'm not sure is true, that probably has strictly provided an incredible ROI as these companies are now valuated in the billions. OTOH, there's a pretty clear through-line between early forecasting research and the rise in popularity of these sites. You can have your own opinion on whether these companies are net-good for the world (the jury's definitely still out), but this is a very significant impact you have to reckon with.
• A lot of people get into the world of rationality and EA through forecasting. This was my entrance into the community. I found competitive forecasting fun, and only later did this give me the exposure to many of the other the things this community cares about -- which I do now as well! Again, hard to quantify the impact of growing the EA/rationality community by ~5%. I'd guess that a few dozen people have taken the Giving Pledge that counterfactually wouldn't have (I know of at least one). Just this alone is an ROI of many millions of dollars.
• AI... not gonna get into this too much, but it's pretty clear that different politicians, policymakers, and influential figures find different arguments appealing. High-level forecasting work is one of several ways of convincing people that AI is something they should take seriously or worry about. Again, quite hard to quantify how much less influence the various sectors of the AI safety lobby would wield right now without the backing of evidence-from-forecasting. Would the AI safety community be worse off without the support of research titans like Hinton and Bengio? Probably. Would they be worse off without a recent popular NYT bestseller? Probably. Would they be worse off without dozens of expert surveys forecasting high chances of negative outcomes? Also, yes, probably they would be.
• Forecasting has been a really good way of getting people who are good at thinking clearly about the future noticed and into good roles! Recruiting is useful.
• Better forecasting infrastructure will help the Dems allocate resources in the 2028 election. In fact, it likely helped the Dems keep the House close in 2024, which has provided an important check on Republican power over the last year or two. Betting markets have outperformed polling aggregators like 538 or the NYT since they took off in popularity and will continue to do so. This will help Democrats allocate funding to tipping point congressional races and is probably worth millions of dollars alone, if not far more (see recent EA focus on democracy).
• Forecasting platforms provide a check on bullshit. It's hard to continually lie when crowdsourced forecasts or prediction markets show a very different story. I think the epistemic environment these days would be even worse than it already is without this. This is similar to the value proposition of Pangram and other AI-detection software in pointing out AI slop. Hard to quantify but certainly valuable.

~~~~~

Additionally, Marcus argues:

If forecasting were really working well and was very useful, you would see the bulk of the money spent not on forecasting platforms but directly on forecasting teams or subsidizing markets on important questions. We have seen very little of this, and instead, we have seen the money go to platforms, tooling, and the like. We already had a few forecasting platforms, the market was going to fund them itself, and yet we continue to create them.

I strongly disagree with this. Hiring teams of encyclopedia writers would have been a far less effective use of money than funding a platform like Wikipedia has been, and I think that's illustrative for understanding why funding forecasting platforms and tooling is wiser than just paying people to make one-off forecasts.

~~~~~

I agree with part of Marcus's article:

I want to propose my leading theory for why forecasting continues to receive 10s of millions per year in funding. That is, it has become a feature of EA/rationalist culture.

I do think that forecasting funding shouldn't justify itself by nature of being some cultural tenet. But has it been?

Coefficient Giving recently closed their forecasting fund, and I don't believe they ever delivered "tens of millions per year in funding." They've funded about $50 million in projects over close to a decade, which would make it ten million per year.

If FRI wants millions of dollars in funding per year going forward, they should probably find a specific way of justifying that. But that's no more or less than is asked of any potential grantee.

The field of forecasting is surprisingly young: it's still in its first generation of researchers. Tetlock and Hanson are still alive and kicking! I personally think there have been a lot of gains to these first couple decades of frantic research, but like for all scientific fields, as it grows in size and age, progress will slow and funding should move to the frontier. As @elifland wrote in a comment to Marcus's post:

But overall I am at least currently much more excited about stuff like AI 2027 or OP worldview investigations than Tetlockian forecasting, i.e. I’m excited about work involving deep thinking and for which the primary value doesn’t come from specific quantitative predictions but instead things like introducing new frameworks (which is why I switched what I was working on).

Obviously funding dumb, boring, old, has-been forecasting stuff is bad. But funding cool, new, cutting-edge, groundbreaking forecasting stuff(good things are good, bad things are bad)...

~~~

...how else would we get important information like this?

cringe zoomer slang voice "That part."

Idk how to measure the hedonic impact of fun forecasting hijinks but it's not zero. Just like all of you very serious LessWrong effort-posters secretly have a lot of fun reading and writing things on here (and get important community good vibes from doing so), Manifold and Metaculus are lowkey two of the coolest internet communities around! Is it worth $5-10 million per year? Ask the 13 year old user on Manifold who just got an interview with Business Insider.

Happy forecasting!

-Ben