gwern comments on [LINK] Get paid to train your rationality - Less Wrong

27 Post author: XFrequentist 03 August 2011 03:01PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (55)
Tags: forecasting tournament training prediction

Comments (55)

You are viewing a single comment's thread. Show more comments above.

Comment author: Morendil 04 August 2011 11:43:04AM 3 points [-]

Apparently the only way to know is to try. It seems likely that there is such a restriction. I'd estimate a better than 70% chance that I get turned down. :)

Comment author: gwern 05 August 2011 08:17:34PM 5 points [-]

I got an email an hour ago from the study saying I was accepted and taking me to the initial survey (a long one, covering calibration on geopolitics, finance, and religion; personality surveys with a lot of fox/hedgehog questions; basic probability; a critical thinking test, the CRT; and then what looked like a full matrix IQ test). The message at the end of all the questions:

Congratulations! You’ve completed the survey. Sometime later this year, we’ll post information on the distribution of answers among those participating in this study.

What comes next? Some of you (by random assignment) will receive an e-mail with a link to a training exercise. Again, we ask you to complete that exercise before forecasting begins on September 1st. That’s the big day for the entire team – the official start of forecasting on 9/1/2011.

Be sure to watch your e-mail for a personalized link to “your” forecasting website. We hope you’re as eager as we are for the tournament to begin.

So I'm marking me as accepted, anyway.

Comment author: Morendil 07 September 2011 10:05:03AM 2 points [-]

And the "tournament" is now begun. Just got email with login instructions.

Looks somewhat similar to PredictionBook, actually. :)

Comment author: gwern 07 September 2011 12:49:06PM 3 points [-]

I did all my predictions last night immediately after the email showed up, so that meant I got to place a lot of bets at 50/50 odds :)

(Then I recorded everything privately in PredictionBook. No point in leaving my predictions trapped on their site.)

Interface-wise, I don't like it at all. I'm still not sure what exactly I am betting at or with, compared to PB with straight probabilities or Intrade with share prices.

Comment author: Morendil 07 September 2011 02:51:23PM 2 points [-]

Did you take the "training refresher"? That includes a general-knowledge test at the end which scores you on both calibration and resolution. My results were pretty poor (but not abysmal):

You got 63% of the items correct, and your average confidence rating over all of the items was 74.33%. (...) In this exercise, your calibration is 11.00 (average confidence minus percent correct). (...) Your confidence when you were correct was 75.26%, and your confidence when you were incorrect was 72.73%. The difference is 2.53%.

I'd be curious to compare with yours if you'd care to share.

Comment author: gwern 07 September 2011 03:00:53PM 2 points [-]

Without actually going through the whole refresher, it seems to be the same; when I did the training, I don't remember that calibration/resolution test. Perhaps that is one of the experimental differences.

Comment author: Morendil 07 September 2011 03:05:02PM 2 points [-]

I didn't remember that test from earlier, either. Worth checking out? I don't mind accidentally unblinding a little if it is an experimental/control difference - curious folks will be curious.

Comment author: gwern 07 September 2011 03:16:52PM 2 points [-]

I just went through the whole thing again; there was no test of that kind at the end. (What there was was the previous multiple-choice quiz about some example forecasts and how they went wrong.) Looks like this is an experimental/control difference. I'd rather not discuss that bit further - this isn't about possibly life-or-death drugs, after all, and I already know where I can find calibration tests like that.

Comment author: Morendil 07 September 2011 03:19:57PM 1 point [-]

Fine with me. :)

BTW, look what I found. Did you know about this one?

Comment author: gwern 07 September 2011 03:34:07PM 1 point [-]

Looks like someone is being very naughty. I've asked him on Twitter.

Comment author: Morendil 08 September 2011 01:38:09PM 1 point [-]

Have you entered any comments on your predictions at the GJ site? (You're supposed to enter a minimum number of comments over one year, and also a minimum number of responses to others' comments. My understanding is that this will in time be run as a team game, with team play conventions.)

From my first experiences, I'm assuming the scoring will be pretty much as with PB.com - based on probability. Their model seems to be calibration/resolution rather than the visual "slope" representation.

Comment author: gwern 08 September 2011 02:21:31PM 1 point [-]

Comments? I don't see any relevant fields for that, checking right now, nor does my 'About' include the substring "comment". Another experimental difference, I guess...

Comment author: Morendil 09 September 2011 07:00:16AM 1 point [-]

The "Why did you answer the way you did" field. I've been assuming we're both using the same underlying app, i.e. Crowdcast. But perhaps we're not...

Powered by Reddit Powered by Reddit