Calibration Test with database of 150,000+ questions

Nanashi

Calibration Test with database of 150,000+ questions — LessWrong

54 Calibration Test with database of 150,000+ questions

by Nanashi

14th Mar 2015

1 min read

54

Hi all,

I put this calibration test together this morning. It pulls from a trivia API of over 150,000 questions so you should be able to take this many, many times before you start seeing repeats.

http://www.2pih.com/caltest.php

A few notes:

1. The questions are "Jeopardy" style questions so the wording may be strange, and some of them might be impossible to answer without further context. On these just assign 0% confidence.

2. As the questions are open-ended, there is no answer-checking mechanism. You have to be honest with yourself as to whether or not you got the right answer. Because what would be the point of cheating at a calibration test?

I can't think of anything else. Please let me know if there are any features you would want to see added, or if there are any bugs, issues, etc.

**EDIT**

As per suggestion I have moved this to the main section. Here are the changes I'll be making soon:

Label the axes and include an explanation of calibration curves.
Make it so you can reverse your last selection in the event of a misclick.

Here are changes I'll make eventually:

Create an account system so you can store your results online.
Move trivia DB over to my own server to allow for flagging of bad/unanswerable questions.

Here are the changes that are done:

Change 0% to 0.1% and 99% to 99.9%
Added second graph which shows the frequency of your confidence selections.
Color code the "right" and "wrong" buttons and make them farther apart to prevent misclicks.
Store your results locally so that you can see your calibration over time.
Check to see if a question is blank and skip if so.

Calibration

Frontpage

54

Mentioned in

17716 types of useful predictions

New Comment

Rendering 0/32 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 3:43 PM

Moderation Log

More from Nanashi

Curated and popular this week

32Comments

Comment Permalink

RowanE11y180

I think the problem here is with many trivia questions you either know the answer or you don't; the dominant factor in my results so far is that I either have no answer in mind, assign 0 probability to my being right and am correctly calibrated there, and then all of my answers at other levels of certainty have turned out right so far so my calibration curve looks almost rectangular.

I might just be getting accurate information that I'm drastically underconfident, but I think this might be one of the worse types of questions to calibrate on. I mean, even if the problem is just that I'm drastically underconfident on trivia questions and shouldn't be assigning less than 50% probability to any of my answers when I have an answer, that sounds sufficiently unrepresentative of most areas where you need calibration, and how most people perform on other calibration tests, for this to be a pretty bad measure of calibration.

Perhaps it would be better as a multiple choice test, so one can have possible answers raised to attention that may or may not be right, and assign probabilities to those?

falenas10811y130

My favorite calibration tools have been one where there was a numerical answer and you had to express a 50% confidence interval, or 90% confidence interval.

Like, a question would be how many stairs are there in the Statue of Liberty? And my 50% interval would be 400-1000, and my 90% interval would be 200-5000.

Looking up the answer it was 354, and I would mark my 50% as wrong and my 90% as right.

1Nanashi11y

That's a fair criticism, but if we're going down this road we've also gotta recognize the limitations of a multiple choice calibration test. Both styles suffer from the "You know it or you don't" dichotomy. If these questions were all multiple choice, you'd still have the same rectangular shaped graph, it would just start at 50% (assuming a binary choice) instead of 0%. The big difference is the solution sets that the different styles represent. There are plenty of situations in life where there are a few specific courses of action to choose from. But there are also plenty of situations where that's not the case. But, I will say that a multiple choice test definitely yields a "pretty" calibration curve much faster than an open-ended test. You've got a smaller range of values, and the nature of it lets you more confidently rule out one answer or the other. So the curve will be smoother faster. Whereas this will be pretty bottom heavy for a while.

9lukeprog11y

0% probability is my most common answer as well, but I'm using it less often than I was choosing 50% on the CFAR calibration app (which forces a binary answer choice rather than an open-ended answer choice). The CFAR app has lots of questions like "Which of these two teams won the Superbowl in 1978" where I just have no idea. The trivia database Nanashi is using has, for me, a greater proportion of questions on which my credence is something more interesting than an ignorance prior.

See in context