Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.
The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.
I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)
+1 Interesting! I've put in a prediction... and also pressed the wrong button on somebody else's prediction (for which the time hasn't elapsed yet) and marked it judged right, hopefully clicking Unknown undoes that...
The advantage of a site like this having been brought to the attention of geeks is that there are at least a few predictions listed to which my answer isn't "how the heck would I know?" :)
Seems like a few other people have been doing the pressing the wrong button thing, if I'm now understanding the user interface correctly? I've tried setting some of those still in the future predictions to unknown, hopefully that's the right thing to do. If so, would it be possible to change the user interface to avoid this error?