Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.
The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.
I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)
Same here - once I entered a percentage, I wasn't sure which button to press, I hesitated between "right" (meaning the percentage I was giving was my confidence that it was right) and "my 2 cents" (which I thought only applieds to when you entered a comment). I selected "right", which was wrong.
The interface needs a bit of polishing.
Me too. The interface for that was confusing enough that I ended up not submitting at all.