gwern comments on [LINK] "Prediction Audits" for Nate Silver, Dave Weigel - Less Wrong

12 Post author: orthonormal 30 December 2011 09:07PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (18)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 09 November 2012 02:19:36AM *  2 points [-]

Jackman only just released his data now (after twittering with me, incidentally, I was able to explain why his R Brier score wasn't matching his hand-calculated Brier score) because he forgot to send it to me last night; and I'm running on fumes - we started this project from scratch yesterday at 5PM and I've been working on it ever since. EDIT: Looks like all the kerfluffle of new Brier/RMSE scores prodded Sam Wang into releasing his precise predictions too! Neat. EDITEDIT: I've gotten Jackman's data, incorporated it, discovered an error in my own data, differed with Jackman, learned he regarded 5 states as such a sure thing he didn't include probabilities while I had simply put in NAs, and now we've converged on his Brier score. Phew! His current Brier score is 0.009713686, a bit worse than Silver's 0.009113725, and both seem to be outperformed by Drew Linzer's 0.003843257. Wang seems to've released the data, but the CSV is unlabeled and I have no idea what half the columns mean...

I'd also like to include a random-guesser equivalent for RMSE... Tomorrow.

Comment author: gwern 09 November 2012 09:05:05PM 0 points [-]

A better Brier random guesser and its RMSE equivalent are now in the R doc and hopefully the blog post will be updated shortly.