I recently wrote a small (python, command-line) program for prediction tracking. Above all, I wanted something that allowed very quick entry, that stored everything in a single plain-text human-readable file on my own computer, and that I could easily customize.

As far as features go, I also wanted a tagging system and to be able to do more sophisticated analysis. For example, I wanted to classify predictions as e.g."work" or "politics" when looking at accuracy, or to see how my calibration changed over different times or with how far in advance I had made a prediction.

Since the predictions that I care most about aren't very interesting to other people, I don't miss the social aspect. Nor do I want to obscure a prediction that contains personal details or store it separately.

An example:

If you wanted to predict that it will rain today, with 50% confidence, and tag the prediction with the "weather" and "external" tags, and be reminded to judge the prediction 2 days from now, you'd type this on the command line:

predict 'rain today' 50 -t weather external -d 2

The created entry in the log will look something like this:

{
"prediction": "rain today",
"confidence": 50,
"notes": null,
"tags": [
"weather",
"external"
],
"date_created": "2014-12-14T20:17:26",
"date_due": "2014-12-16T20:17:26",
"state": "open"
}

Running "predict --due" will either ask you about any predictions that are ready to be judged and update the corresponding log entry, or just change "state" to "due" so you can ctrl-f and edit the log directly. (Since the log is just text, you can manually edit any entry, like changing a prediction from "open" to "true" or "false").

Running "predict --stats" will dump statistics.

I'm very happy with this work flow, and I'm hoping to clean up the code and share it when I get a moment. It's all of ~250 lines, but I couldn't find what I wanted out there already.