Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.
The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.
I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)
Just out of curiosity, are you a startup, a non profit or a guy doing a side project?
I predict the site's userbase will not explode overnight but will escalate in the shape of a hockey stick. That's how these things usually happen. You will have to keep improving it even while the userbase is still low, otherwise people will think the site is dying and they will stop showing up. Interesting things need to already be happening on the site before a larger audience will keep coming back to it, not vice versa.
Also, you need to add documentation no matter how simple and intuitive you think the sites features are. They don't seem as intuitive from the outside. By 'documentation' I mean a short and EXPLICIT description of what each feature does. I like the 'help' button near the timeframe for the prediction. You could add help buttons next to everything. Also a faq would be nice.
Overall I think the site has great potential. Keep up the good work.
We're Investling, which is a handfull of startups and an IT consultancy. We're for-profit, with some non-profit projects on the side (in part because we'll make more profits if we can help save the world from surprise conversion to paperclips). The majority of our non-profit work is SIAI related.
Some projects follow that pattern. Some projects never hockey-... (read more)