Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.
The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.
I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)
Whether they add features will depend on whether people seem interested in using it, they say.
Official answer: Eliezer's right. If we see traffic growing we'll invest in further development.
We can think of many things we could do to make the site better… but those users who currently use it don't use it enough, and if they tell their friends about it their friends don't become regular users (often enough).
Hosting the current code is very cheap and easy, so the site's in little danger of being shut down, but we won't be developing it further unless you guys and gals (and your friends, and their friends) pile on the love.