Probability theory does not extend logic (predicate calculus). In particular, freely mixing logical quantifiers (∀, ∃) and probability statements gets messy fairly quickly, and the tools to disambiguate their meaning may not be found solely in probability theory (but perhaps in statistical inference or in the study of causality.)
The original article made it sound like that was an area of unfinished research (at the time it was written). If that's been solved, I imagine the original writer might want to know about it.
No, this hasn't been solved. But I imagine that mixing logical quantifiers and probability statements would be less messy if one e.g., knows the causal graph of the events to which the statements refer. This is something that the original post didn't mention, but which I thought was interesting.
Highlights
Index
Sign up here or browse past newsletters here.
Prediction Markets & Forecasting Platforms
Augur
Augur has launched Augur Turbo on Polygon, the same second-layer chain which also hosts Polymarket. The website to access it is hosted on IPFS, hence the long address when one accesses it through a portal.
To unpack the tech stack:
So far, Augur Turbo merely has NBA markets, with low volume, and low liquidity.
From Augur’s early history, Is Augur Being Gamed explains what exactly Poyo did to profit from the creation of invalid markets. Back in the day, Augur didn’t have a tradable “invalid” resolution. Instead, invalid or ambiguous markets were resolved 50-50. This allowed Poyo to profit by creating markets he knew would resolve as invalid, and then buying the cheaper side.
CSET-Foretell
CSET-Foretell has added the ability for users to suggest questions.
Good Judgment training for “Foretell Pros”—the best scoring forecasters during CSET-Foretell's first season—continues. Because Foretell Pros might get culled if they perform worse than the crowd, and because their score is proportional to their difference from the crowd, they have an incentive not to share information. When this was pointed out, CSET-Foretell answered with an impassioned appeal to the better angels of our nature. It seems it worked to some extent, and participants are sharing more of their reasoning within the platform and community.
Good Judgment Inc/Good Judgment Open
Per the Good Judgment Open Newsletter, product lead and Superforecaster Luis Enrique Urtubey De Césaris has some openings in his office hours coming up on June 11th and 18th. The contact email provided is beta@goodjudgment.com, no schedule is given.
The Financial Times reports on Superforecaster predictions for "When will the number of doses administered globally reach 5 billion?"
How to find out your calibration on Good Judgment Open and CSET-Foretell
Misha Yagudin, friend of the newsletter and fellow Samotsvety Forecasting team member on CSET-Foretell, has programmed a site which allows users to get their calibration chart for their predictions on Good Judgment Open and other Cultivate Labs platforms, such as CSET-Foretell. See this one-minute video for how to use it. Using Misha’s site, my calibration chart looks as follows:
This means that I'm under-confident around the 15% (resp. 85%) level. I know why this is: I was assigning a 15% chance to questions which gave me the feeling that "this is most likely not the case, but I’m not completely sure.” As it turns out, the kinds of questions on Good Judgment Open which generate that feeling instead happen around 10% of the time.
For comparison, here is the historical calibration of Canadian strategic intelligence forecasts, which I calculated using this dataset:
And here is the calibration of Spock, from Julia Galef’s The Scout Mindset (h/t Gavin Leech, Michał Dubrawski):
Here are some hypothesis about why Spock's calibration is skewed:
Hypermind
Hypermind has been experimenting with new methods of eliciting, incentivizing and scoring long-range forecasts. The first mechanism consists of “drip rewards.” In short, if you want to get predictions about an event in 2030, you could try to promise forecasters a reward in 2030. But they might not find it very motivating. Instead, you could ask the same question each year (2021, 2022,...) until 2030, and reward forecasters according to how much their prediction one year resembles the crowd's predictions in the next year.
Here is a summary of the mechanism which includes more twists, such as making the reward time random, and increasing rewards as resolution time approaches. Hypermind is trying out this method for predicting COVID-19 vaccinations by 2029, with a price pool of $30,000.
Comments from colleagues centered around the fact that Hypermind wants to patent the method, but there is plenty of prior art. For instance, in machine learning a similar idea is known as Temporal difference learning (h/t Misha Yagudin.)
The second method is more speculative. Forecasters make an object-level prediction, and a meta-prediction on what the crowd prediction will be. Then, forecaster predictions are adjusted—based on the meta-predictions—to increase the probability of "surprisingly popular" predictions. See Wikipedia on the Surprisingly popular method for a simplified example.
The paper from which this method comes considered forecasts on discrete bins. Per its public writeup, Hypermind applies this method to predicting continuous distributions by dividing continuous distributions into discrete bins and then ignoring bins with probabilities below 5%!:
Hypermind tried this method for predicting the state of AI in 2030 where ignoring events which have a lower than 5% probability seems like a particularly bad idea, given that those events might be particularly impactful.
Metaculus
GlobalGuessing interviews Gaia Dempsey, Metaculus’ CEO, and continues analyzing Metaculus questions.
SimonM kindly curated the top comments from Metaculus this past May. They are:
A tournament on Virginia COVID-19 cases was also covered by a quaint local Virginian newspaper.
Polymarket
Polymarket featured plenty of markets about NBA playoffs. They also sponsored GM Hiraru Nakamura's Twitch stream throughout the #FTXCryptoCup, a chess tournament organized by FTX. As part of their sponsorship, they gave away $20 to 500 new users; it seems like the link is still up.
Polymarket's microgrants program spawned Polystats, which displays statistics about markets. The site might make it easier for liquidity providers to choose where to stake their funds, and competes with an earlier site, PolymarketWhales.
"Sandwiching" bots, covered in the previous edition of this newsletter, continue to be an annoyance.
As for markets, Will the 2021 Tokyo Olympics take place? is sitting at ~81% (~77% on FTX) (!), and Will Joe Biden be President of the USA on September 30, 2021? is currently sitting at ~92% (!?).
PredictIt
Old Bull TV is a Youtube Channel which covers PredictIt markets. Their episode When PredictIt Met Kevin presents the case of Kevin Paffrath, a random influencer with 1.63M Youtube followers who got his followers and associates to buy his shares for the Who will be the governor of California on Dec. 31? market.
In the News
How the U.S. Government Can Learn to See the Future argues that rigorous probabilistic forecasting, keeping score of assessments, and employing the “wisdom of crowds” would lead to better US intelligence assessments. They also point out that forecasting projects did not survive the “valley of death”—the space between being a pilot program and being an established product for the Department of Defense which many initiatives fail to cross.
There has been some recent brouhaha in the news (archive link) about whether COVID-19 originated from a lab. The issue was previously featured in the January edition of this newsletter:
Gauging for disasters: Neighbor shares distrust of river forecasting following flood event gives a slice-of-life picture of how forecasting affects common folks. On the one hand, the interviewee is probably suffering from hindsight bias. But on the other hand, it does seem like the forecasts were not robust to further rainfall, and that grizzled grumpy locals might have had more information than the forecasters.
Blog Posts
Probability theory does not extend logic (predicate calculus). In particular, freely mixing logical quantifiers (∀, ∃) and probability statements gets messy fairly quickly, and the tools to disambiguate their meaning may not be found solely in probability theory (but perhaps in statistical inference or in the study of causality.)
My attempt to think about AI timelines, by Ben Snodin, gives his probabilities for AI timelines based on a combination of inside and outside views, after thinking about it for 40 hours.
Data on forecasting accuracy across different time horizons and levels of forecaster experience, by Charles Dillon, builds on earlier work by niplav. The post might be useful to individual forecasters seeking to learn about past failure modes when forecasting long-range questions.
Predict responses to the "existential risk from AI" survey (also on LessWrong):
Papers
Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model
Hard to categorize
The Miami International Securities Exchange (MIAX) was set to offer corporate tax futures on the Minneapolis Grain Exchange (MGEX). The site currently seems to be down, but a copy remains on the Internet Archive.
The United States' distopically named National Institute of Justice has a Recidivism Forecasting Challenge, with a total prize pool of $723,000, divided across many categories. I imagine that the student and small team categories should be reasonably accessible, but “individuals must be U.S. residents and companies must have an office with a U.S. business license.”
The epiforecast group at the London School of Hygiene & Tropical Medicine opened the UK Covid-19 Crowd Forecasting Challenge, with a prize pool of £175.
OpenPhilanthropy has a forecasting-related job offer for a relatively junior role:
Long Content
Imprecise probability is an attempt to generalize probability theory to allow for uncertainty about or multiplicity of probability estimates. For example, consider expressing one's uncertainty by giving the odds you'd be willing to take in favor of X, and the odds you'd be willing to take against X, but those odds having a spread. See also this summary by Ben Snodin of two abstruse ivory-tower papers about the topic.
Anthropics: different probabilities, different questions dissolves the apparent paradox that different anthropic theories give different probabilities to the same event.
The economics of faith: using an apocalyptic prophecy to elicit religious beliefs in the field:
Note to the future: All links are added automatically to the Internet Archive. In case of link rot, go there and input the dead link.