Maintaining a vegetable garden gives plenty of opportunities for such quick predictions, even for sequentially updated ones. 'I have to weed the carrot rows. But I have only done them last week! Surely there will be less than a hundred weed seedlings?' -> 'Oh, this is how it looks today! Okay, raising the estimate to 150.' -> 'My back! Why do I grow carrots, anyway? It's a carpet of weeds!':)

Apart from PredictionBook, what's out there in terms of software for tracking your predictions?

(PredictionBook is nice, but there are predictions I make that I don't want to have online, and I find the inability to organise predictions rather inconvenient. Ideally, some kind of system with tags would be nice. I guess I could just set up a SQLite database and write some shell scripts to add/remove data, but something neater would be nice.)

Microsoft Outlook Business Contact Manager provides ways forward to utilising prediction. Within its Task scheduling one has opportunity to estimate what percentage of the set task is already completed. Also how long the task will take is estimated by the user.

I find B.C.M highly useful for focusing prediction and task achievement.

Last year I built a spreadsheet for helping with #1 (Predict how long a task will take you) which I still use daily.

Duration Calibration Template

You input the name of a task (with categorization if you want) and a lower and upper bound for how long you expect it to take. Start a timer, do the thing, then put in the actual time. If you were within the bounds, you get a ✓ in the next column, otherwise you get your error, green for overestimate and red for underestimate. I'll probably add some stuff that does aggregate stats on how often you're within the bounds (i.e. what actual confidence interval those bounds represent for you) but for now it's mostly just the experience of doing it that's helpful.

I think I've become better calibrated over months of using this, although some days it's way off because I don't give it enough attention. There's a pretty big difference between a duration estimate made in 1 second vs 5 seconds.

When I first tried this, I thought that it would be annoying, but I actually found it to be a really enjoyable way to do lots of different kinds of things, because it kept me really focused on the specific thing I was timing, rather than going off on other tangents that felt productive but were unfocused, or jumping around. In addition to tracking "tasks", I would also sometimes track things like reading a chapter of hpmor, or how long it would take me to go get a snack.

(I was planning to write a top-level post about this, but since it's super topical I figured I might as well post it as a comment here. Top level post to follow, perhaps.)

Does making a lot of predictions cause any stress? To me it does, almost like a sense that I'm always "on duty". Also, I'm a bit easily distracted and it could distract me from the task at hand. So I try to strike some sort of balance. Thoughts?

Also, sometimes predictions negatively affect the amount of enjoyment I get. Ex. I'm a big basketball fan and I do love making predictions in basketball, but sometimes it isn't fun to have predicted that the team I like is going to lose.

I agree with your closing comments, but I think it's useful to make the distinction that it's helpful to begin with predictions that are very clearly successful or unsuccessful, even if this might be construed as erring toward more 'objective' metrics. This is why I chose #13 to begin forming my predicting habit; my paraphrases of others' positions are usually easily categorized as either correct or incorrect and salient outcomes are better for habit formation.

Great post!

Certainly, for me, there has always been a gap between how useful I think predictions are, in theory, and how often I make them.

Same here.

There's probably some additional value you can extract by writing down quantitative confidence levels, and by devising objective metrics that are impossible to game, rather than just relying on your subjective impressions.

I agree.

But in most cases I don't think that additional value is worth the cost you incur from turning predictions into an onerous task.

I disagree in that (1) I think much of the value of predictions would come from the ability to examine and analyze my past prediction accuracy and (2) I don't think the task of recording the predictions would necessarily be very onerous (e.g. especially if there is some recurring prediction which you don't have to write a new description for every time you make it).

I really like Prediction Book (which I just checked out for the first time before Googling and finding this post), but it doesn't offer sufficient analysis options to make me want to really begin using it yet.

But this could change!

I would predict (75%) that I would begin using it on a daily basis (and would continue to do so indefinitely upon realizing that I was indeed getting sufficient value out of it to justify the time it takes to record my predictions on the site) if it offered not just the single Accuracy vs 50-100% Confidence plot and graph, but the following features:

• Ability to see confidence and accuracy plotted versus time. (Useful, e.g. to see weekly progress on meeting some daily self-imposed deadline. Perhaps you used to meet it 60% of days on average, but now you meet it 80% of days on average. You could track your progress while seeing if you accurately predict progress as well, or if your predicted values follow the improvement.)

• Ability to see 0-100% confidence on statistics plot, instead of just 50-100%. (Maybe it already includes 0-50% and just does the negative of each prediction (?). However, if so, this is still a problem since I may have different biases for 10% predictions than 90% predictions.)

• Ability to set different different prediction types and analyze the data separately. (Useful for learning how accurate one's predictions are in different domains.)

• Ability to download all of one's past prediction data. (Useful if there is some special analysis that one wants to perform.)

• A public/private prediction toggle button (Useful because there may be times when it's okay for someone to hide a prediction they were embarrassingly wrong about or someone may want to publicize a previously-private prediction. Forcing users to decide at the time of the prediction whether their prediction will forever be displayed publicly on their account or remain private forever doesn't seem very user-friendly.)

• Bonus: An app allowing easy data input when not at your computer. (Would make it even more convenient to record predictions.)

Some of these features can be achieved by creating multiple accounts. And I could accomplish all of this in Excel. But using multiple accounts or Excel would make it too tedious to be worth it. The value is in having the graphs and analysis automatically generated and presented to you with only a small amount of effort needed to input the predictions in the first place.

I don't think any of these additional features would be very difficult to implement. However, I'm not a programmer, so for me to dive into the Prediction Book GitHub and try to figure out how to make these changes would probably be quite time-consuming and not worth it.

Maybe there is someone else who agrees that these features would be useful to them who is a programmer and would like to add some or all of the suggested features I mentioned? Does anyone know the people who did most of the work programming the current website?

You could always play ask a stranger a question.

I like this post. I lean towards skepticism about the usefulness of calibration or even accuracy, but I'm glad to find myself mostly in agreement here.

For lots of practical (to me) situations, a little bit of uncertainty goes a long way concerning how I actually decide what to do. It doesn't really matter how much uncertainty, or how well I can estimate the uncertainty. It's better for me to just be generally humble and make contingency plans. It's also easy to imagine that being well-calibrated (or knowing that you are) could actually demolish biases that are actually protective against bad outcomes, if you're not careful. If you are careful, sure, there are possible benefits, but they seem modest.

But making and testing predictions seems more than modestly useful, whether or not you get better (or better calibrated) over time. I find I learn better (testing effect!) and I'm more likely to notice surprising things. And it's an easy way to lampshade certain thoughts/decisions so that I put more effort into them. Basically, this:

Or in other other words: the biggest problem with your predictions right now is that they don't exist.

To be more concrete, a while back I actually ran a self-experiment on quantitative calibration for time-tracking/planning (your point #1). The idea was to get a baseline by making and resolving predictions without any feedback for a few weeks (i.e. I didn't know how well I was doing--I also made predictions in batches so I usually couldn't remember them and thus target my prediction "deadlines"). Then I'd start looking at calibration curves and so on to see if feedback might improve predictions (in general or in particular domains). It turned out after the first stage that I was already well-calibrated enough that I wouldn't be able to measure any interesting changes without an impractical number of predictions, but while it lasted I got a moderate boost in productivity just from knowing I had a clock ticking, plus more effective planning from the way predictions forced me to think about contingencies. (I stopped the experiment because it was tedious, but I upped the frequency of predictions I make habitually.)

To the best of my knowledge, human brain is a simulation machine. It unconsciously making prediction about what sensory input it should expect. This include the higher level input, like language and even concepts. This is the basic mechanism underlying surprise and similar emotion. Moreover, it only makes simulation on the things it cares about and filter the rest.

Given this, I would think that most of your prediction is obsolete, because we are doing this unconsciously. Example:

1. You predict you will finish the task one week early. But you are ended up finishing one day early. You are not surprised. But if you ended up finishing one day late, then you would be surprised. When people are surprised by the same trigger often enough, most normal people I presume, will update their believe. I know this is related to planning fallacy, but I think my arguments still hold water.

2. You post a post on Facebook. You didn't make any conscious prediction on the reaction of the audience. You got one million likes. I bet you will be surprised and scratching your mind about why and how you could get such reaction.

Otherwise, I still see some value in what you are doing, but not because of prediction per se, but because you it effectively mitigate bias. For example. "Predict how well you understand someone's position by trying to paraphrase it back to him." It addresses illusion of transparency. But I think there is not much more value in making prediction rather than simply making a habit to paraphrase more often than otherwise without making prediction.

Making conscious prediction, on top of the unconscious one, is cognitively costly. I do think it might improve one's calibration and accuracy and is superior to the improvement made by the surprise mechanism alone. However, the question is, is the calibration and accuracy improvement worth the extra cognitive cost?

I'm not convinced that improving calibration will not improve accuracy because predictions are often nested within other predictions. For example, suppose we are trying to make a prediction about P, and the truth or falsity of Q, R and S are relevant to the truth of P in some respect. We might use as a basis for guessing P that we are ninety five percent confident in our guesses about Q, R & S, (suppose the truth of all three would guarantee P). Now suppose we become less confident through better calibration and decide there is only a 70% chance that Q, a 70% chance that R and a 70% chance that S, leading to a compound probability of less <50%. Thus overall accuracy can be improved by calibration.

My hunch is that encouraging people that have to manage an unpredictable or tricky health condition to predict and note their prediction of how good or bad an activity will be for their pain / energy / / mood / whatever else would be a very useful habit that both frees people up to do things and prevents them from doing too much of what hurts. Julia, have you or anyone from CFAR looked at partnering with a pain management or other type of disease management team or setting to see how many of the rationality skills would be helpful?

Cards Against Humanity is a party game that can help develop the ability to predict what each person in the group finds funny, or which answer they will choose. If you get better at predicting, you get better at the game. I mention it as a fun way to get better at predicting.

There the possibility to write down predictions whether you make an agreement with another person. If I schedule a date then I can predict the chance of flaking.

That means that I stop fully trusting the person to be there. I treat them more like a number than as a person.