Wits and Wagers is apparently a board game, where players compete to be well-calibrated with respect to their trivia knowledge. I haven't played it.

Has someone else here played it? If so, what was your experience? Would it be good rationalist/bayesian training?

In my previous post, I alluded to a result that could potentially convince a frequentist to favor Bayesian posterior distributions over confidence intervals. It’s called the complete class theorem, due to a statistician named Abraham Wald. Wald developed the structure of frequentist decision theory and characterized the class of decision rules that have a certain optimality property.

Frequentist decision theory reduces the decision process to its basic constituents, i.e., data, actions, true states, and incurred losses. It connects them using mathematical functions that characterize their dependencies, i.e., the true state determines the probability distribution of the data, the decision rule maps data to a particular action, and the chosen action and true states together determine the incurred loss. To evaluate potential decision rules, frequentist decision theory uses the risk function, which is defined as the expected loss of a decision rule with respect to the data distribution. The risk function therefore maps (decision rule, true state)-pairs to the average loss under a hypothetical infinite replication of the decision problem.

Since the true state is not known, decision rules must be evaluated over all possible true states. A decision rule is said to be “dominated” if there is another decision rule whose risk is never worse for any possible true state and is better for at least one true state. A decision rule which is not dominated is deemed “admissible”. (This is the optimality property alluded to above.) The punch line is that under some weak conditions, the complete class of admissible decision rules is precisely the class of rules which minimize a Bayesian posterior expected loss.

(This result sparked interest in the Bayesian approach among statisticians in the 1950s. This interest eventually led to the axiomatic decision theory that characterizes rational agents as obeying certain fundamental constraints and proves that they act as if they had a prior distribution and a loss function.)

Taken together, the calibration results of the  previous post and the complete class theorem suggest (to me, anyway) that irrespective of one's philosophical views on frequentism versus Bayesianism, perfect calibration is not possible in full generality for a rational decision-making agent.

I'm planning a two-part sequence with the aim of throwing open the question in the title to the LW commentariat. In this part I’ll briefly go over the concept of calibration of probability distributions and point out a discrepancy between calibration and Bayesian updating.

It's a tenet of rationality that we should seek to be well-calibrated. That is, suppose that we are called on to give interval estimates for a large number of quantities; we give each interval an associated epistemic probability. We declare ourselves well-calibrated if the relative frequency with which the quantities fall within our specified intervals matches our claimed probability. (The Technical Explanation of Technical Explanations discusses calibration in more detail, although it mostly discusses discrete estimands, while here I'm thinking about continuous estimands.)

Frequentists also produce interval estimates, at least when "random" data is available. A frequentist "confidence interval" is really a function from the data and a user-specified confidence level (a number from 0 to 1) to an interval. The confidence interval procedure is "valid" if in a hypothetical infinite sequence of replications of the experiment, the relative frequency with which the realized intervals contain the estimand is equal to the confidence level. (Less strictly, we may require "greater than or equal" rather than "equal".) The similarity between valid confidence coverage and well-calibrated epistemic probability intervals is evident.

This similarity suggests an approach for specifying non-informative prior distributions, i.e., we require that such priors yield posterior intervals that are also valid confidence intervals in a frequentist sense. This "matching prior" program does not succeed in full generality. There are a few special cases of data distributions where a matching prior exists, but by and large, posterior intervals can at best produce only asymptotically valid confidence coverage. Furthurmore, according to my understanding of the material, if your model of the data-generating process contains more than one scalar parameter, you have to pick one "interest parameter" and be satisfied with good confidence coverage for the marginal posterior intervals for that parameter alone. For approximate matching priors with the highest order of accuracy, a different choice of interest parameter usually implies a different prior.

The upshot is that we have good reason to think that Bayesian posterior intervals will not be perfectly calibrated in general. I have good justifications, I think, for using the Bayesian updating procedure, even if it means the resulting posterior intervals are not as well-calibrated as frequentist confidence intervals. (And I mean good confidence intervals, not the obviously pathological ones.) But my justifications are grounded in an epistemic view of probability, and no committed frequentist would find them as compelling as I do. However, there is an argument for Bayesian posteriors over confidence intervals than even a frequentist would have to credit. That will be the focus of the second part.

One of my favorite fiction genres is alternative history.  The basic idea of alternative history is to write a story set in an alternate universe where history played out differently.  Popular alternate histories include those where the Nazis win World War II, the USSR wins the Cold War, and the Confederate States of America win the American Civil War.  But most of the writing in this genre has a serious flaw:  the author starts out by saying "wouldn't it be cool to write a story where X had happened instead of Y" and then works backwards to concoct historical events that will lead to the desired outcome.  No matter how good the story is, the history is often bad because at every stage the author went looking for a reason for things to go his way.

Being unsatisfied with most alternate histories, I like to play a historical "what if" game.  Rather than asking the question at the conclusion, though (like "what if the Nazis had won the war"), I ask it at an earlier moment, ideally one where chance played an important role.  What if Napoleon had been convinced not to invade Russia?  What if the Continental Army had not successfully retreated from New York?  What if Viking settlements in Newfoundland had not collapsed?  These are as opposed to "What if Napoleon had never been defeated?", "What if the Colonies had lost the American Revolutionary War?", and "What if Vikings had developed a thriving civilization in the Americas?".  I find that replaying history in this way a fun use of my analytical skills, but more importantly a good test of my rationality.

One of the most difficult things in thinking of an alternative history is to stay focused on the facts and likely outcomes.  It's easy to say "I'd really like to see a world where X happened" and then silently or overtly bias your thinking until you find a way to achieve the desired outcome.  Learning to avoid this takes discipline, especially in a domain like alternate history where there's no way to check if your reasoning turned out to be correct.  But unlike imagining the future, making an alternate history does have the real history to measure up against, so it provides a good training ground for futurist who don't want to wait 20 or 30 years to get feedback on their thinking.

Given all this, I have two suggestions.  One, this indicates that a good way to teach history and rational thinking at the same time would be to present historical data up to a set point, ask students to reason out what they think will happen next in history, and then reveal what actually happened and use the feedback to calibrate and improve our historical reasoning (which will hopefully provide some benefit in other domains).  Second, a good way to build experience applying the skills of rationality is publicly present and critique alternate histories.

In that vein, if there appears to be sufficient interest, I'll start doing a periodic article here dedicated to the discussion of some particular alternative history.  The discussion will be in the comments:  people can propose outcomes, then others can revise and critique and propose other outcomes, continuing the cycle until we hit a brick wall (not enough information, question asks something that would not have changed history, etc.) or come to a consensus.

What do you all think of this idea?