Like a collector, I gather resources and information. Storing it and categorizing it. I do this for any project I undertake. But for Forecasting AI Futures, I realized the collection might be quite useful for others.
So, I set to work on restructuring it and make it look nice!
The resource hub is still in its infancy but will grow to become more comprehensive over time. If you have ideas for how to improve it—helping it mature into its adult form—they are very welcome.
There are two major parts. The Prediction Database:
And the Forecasting AI Ecosystem:
I’ve also included some of the resources in this post.
AI trends: Analyzing trends in AI development and investments is crucial for forecasting AI. There is an organization doing exactly this: Epoch AI. See their collection of Machine Learning Trends.
Prediction accuracy and calibration: Blindly deferring to a prediction without verifying its reliability is unwise. Check calibration and accuracy (e.g. Brier score). See Calibration City for calibration and accuracy data for popular prediction markets and Metaculus. Credible forecasters may publish their track records, like this track record for the Samotsvety forecasting group.
Forecasting applications: Forecasting is a powerful decision-making tool. For instance, Convergence Analysis combines AI scenario forecasting with governance research and recommendations (see their Theory of Change).
Forecasting bots: One AI capability of particular interest to forecasters is the ability to generate accurate predictions. AIs are, as far as I know, still lacking in forecasting skills compared to professional human forecasters. I have included some information, predictions, and resources regarding AI forecasting agents in the resource hub.
The prediction database
The current space of predictions
The prediction database highlights where forecasting communities are focusing their attention, and what they are missing. It is a tool for discovering questions that have yet to be asked—and then asking them.
Why a hierarchical structure?
A spreadsheet would have been more flexible than the current hierarchical structure of the database—each prediction could have several tags. It would, however, be harder to include important information in the database, such as links to benchmark leaderboards and other resources. Additionally, I find a hierarchical structure more intuitive—it mirrors my way of thinking about information and forecasting.
Selected predictions
Not every AI-related prediction is included in the database. The ones included reflect what I find meaningful and interesting. I have, however, surely introduced some pointless or vague predictions as well—I have not been extremely prudent with the selection since the database was originally created for personal use. I plan to refine the selections over time.
Like a collector, I gather resources and information. Storing it and categorizing it. I do this for any project I undertake. But for Forecasting AI Futures, I realized the collection might be quite useful for others.
So, I set to work on restructuring it and make it look nice!
Here is the result: Forecasting AI Futures Resource Hub
The resource hub is still in its infancy but will grow to become more comprehensive over time. If you have ideas for how to improve it—helping it mature into its adult form—they are very welcome.
There are two major parts. The Prediction Database:
And the Forecasting AI Ecosystem:
I’ve also included some of the resources in this post.
Key Links
Some remarks
The prediction database
The current space of predictions
The prediction database highlights where forecasting communities are focusing their attention, and what they are missing. It is a tool for discovering questions that have yet to be asked—and then asking them.
Why a hierarchical structure?
A spreadsheet would have been more flexible than the current hierarchical structure of the database—each prediction could have several tags. It would, however, be harder to include important information in the database, such as links to benchmark leaderboards and other resources. Additionally, I find a hierarchical structure more intuitive—it mirrors my way of thinking about information and forecasting.
Selected predictions
Not every AI-related prediction is included in the database. The ones included reflect what I find meaningful and interesting. I have, however, surely introduced some pointless or vague predictions as well—I have not been extremely prudent with the selection since the database was originally created for personal use. I plan to refine the selections over time.