x

LESSWRONG
LW

Tom Liptay

Message

12

3y

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

In Q4 we ran the second tournament in the AI Benchmarking Series which aims to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 44 bots compete for $30,000 on 402 questions with a team...

Feb 19, 202514

Tom Liptay

Subscribe

Message

12

3y

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

In Q4 we ran the second tournament in the AI Benchmarking Series which aims to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 44 bots compete for $30,000 on 402 questions with a team...

Feb 19, 202514

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

Molly

Molly, Tom Liptay

1y

In Q4 we ran the second tournament in the AI Benchmarking Series which aims to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 44 bots compete for $30,000 on 402 questions with a team of ten Pros serving as a human benchmark on 122 of those 402. We found that:

Metaculus Pro Forecasters were better than the top bot “team” (a team of one, this quarter), but not with statistical significance (p = 0.079) using log scoring with a weighted t-test.
Top bot performance improved relative to the Pro benchmark to a -8.9 head-to-head score in Q4 2024,

... (read 3884 more words →)

14