gwern comments on [LINK] Get paid to train your rationality - Less Wrong

27 Post author: XFrequentist 03 August 2011 03:01PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (55)
Tags: forecasting tournament training prediction

Comments (55)

You are viewing a single comment's thread. Show more comments above.

Comment author: Morendil 07 September 2011 02:51:23PM 2 points [-]

Did you take the "training refresher"? That includes a general-knowledge test at the end which scores you on both calibration and resolution. My results were pretty poor (but not abysmal):

You got 63% of the items correct, and your average confidence rating over all of the items was 74.33%. (...) In this exercise, your calibration is 11.00 (average confidence minus percent correct). (...) Your confidence when you were correct was 75.26%, and your confidence when you were incorrect was 72.73%. The difference is 2.53%.

I'd be curious to compare with yours if you'd care to share.

Comment author: gwern 07 September 2011 03:00:53PM 2 points [-]

Without actually going through the whole refresher, it seems to be the same; when I did the training, I don't remember that calibration/resolution test. Perhaps that is one of the experimental differences.

Comment author: Morendil 07 September 2011 03:05:02PM 2 points [-]

I didn't remember that test from earlier, either. Worth checking out? I don't mind accidentally unblinding a little if it is an experimental/control difference - curious folks will be curious.

Comment author: gwern 07 September 2011 03:16:52PM 2 points [-]

I just went through the whole thing again; there was no test of that kind at the end. (What there was was the previous multiple-choice quiz about some example forecasts and how they went wrong.) Looks like this is an experimental/control difference. I'd rather not discuss that bit further - this isn't about possibly life-or-death drugs, after all, and I already know where I can find calibration tests like that.

Comment author: Morendil 07 September 2011 03:19:57PM 1 point [-]

Fine with me. :)

BTW, look what I found. Did you know about this one?

Comment author: gwern 07 September 2011 03:34:07PM 1 point [-]

Looks like someone is being very naughty. I've asked him on Twitter.

Comment author: Morendil 07 September 2011 05:19:38PM 2 points [-]

I've been wondering if it's not the other way round, the Good Judgement project copying Inkling Market's questions? What info do you have that leads you to think the copying was in the direction you assume?

My evidence for the other way round is that the Brent question has a starred footnote which is present on IM but not on GJ, while the star is in the text of the GJ question.

Comment author: gwern 07 September 2011 08:27:19PM 2 points [-]

The description, from the latest email, is

First, for those of you who logged in before September 6, please be aware that the tournament's sponsor issued 10 new questions today, which we have posted. As the About page notes, new questions usually will be distributed on Mondays (but not every Monday). These questions arrived on Tuesday because of the Labor Day holiday.

My understanding was that the sponsor was IARPA. And googling, I don't see any listed connections between Inkling and Good Judgement Project.

Comment author: Douglas_Knight 07 September 2011 10:14:54PM 1 point [-]

Stray asterisks are very suspicious. I see one in the Inkling question, but I don't see the footnote itself. It has a "background information" section, but all their questions do. Is "last price" a technical term? If the usual term is "settlement price," and GJ doesn't make that clear, then it is quite suspicious.

Here are two two more Inkling questions with asterisks. One has an explicit footnote. The other is a change to the question.

Powered by Reddit Powered by Reddit