My comment from July 5, "Go Bayes! So if you just make your priors big enough, you never have to change your mind.", was rather snarky, but it illustrates a real problem. If your priors are not reasonably accurate, it takes a lot of new information and updating to get it straightened out. That is one reason a lot of introductions to Bayes rule use medical decision making which has reasonably well-established base-rates (priors) to begin with.

Off-topic, but in the context of "best mistake", here is John Ringo's definition of serendipity from The Last Centurion:

We were saved by serendipity. (Which is a term meaning "I fucked up but things came out better than if I hadn't.")

Derek Lowe also commented on the studies. Repeating my comment there:

So the comparison of the two experiments shows that underfeeding results in life extension over monkeys that over-eat, but not over monkeys that eat a normal diet. Where is the surprise there?

ADDED: I just noticed the paragraph here is missing a key bit of information needed to make sense of my comment. The WNPRC experiment, which found positive results from calorie restriction, fed their controls ad libitum, as much as they wanted to eat. The newer NIA experiment fed the controls a standard, healthy diet, and found no effect of diet restriction.

I'm very skeptical of reasoning like "it was like that in ancestral environment so it must be good". There are at least three reasons that makes me uncomfortable with the reasoning :

1. Even if we consider evolution to be a perfect optimizer (which it is not), there is a huge difference between "our digestion system is optimized to make the best possible use of food X" and "food X is best possible food for our digestion system". If you made an algorithm A optimized to transmit data on a noisy channel N, it doesn't mean the algorithm wouldn't run better on a less noisy channel C. There may be an algorithm B that work better on the clear channel C than A, but still, A can work better on C than on N.

2. Evolution doesn't optimize for the same purpose we do. Evolution doesn't optimize for us to live long, it has a very low pressure to make us live past ~60, for example.

3. We have completely different lifestyles and activities than we did during paleolithic. And the optimal diet very likely depends of lifestyle and activities.

That said, what would convince me to do a diet is not a plausible-sounding reasoning, but some evidence of short-term and long-term effects on a sane sample size, with a control group. Something which seems very rare in the diet field, saddly.

I agree that the only way to practice decisions is to make them, but I think there is more to it than that. The deliberate part of deliberate practice is that you are actively trying to get better. The deliberate performance paper I linked to touches on this a bit, in that deliberate practice is challenging for professionals and that something else might work better (they advocate the first 5 methods in that paper).

Beyond making decisions, you need to have an expectation of what will happen, otherwise hindsight bias is that much harder to overcome. It's the scientific method: hypothesis->test->new hypothesis. Without defining what you expect ahead of time, it is much easier to just say "Oh yeah, this makes sense" and normalize without actually improving understanding.

An alternative to improving your intuition and removing your biases would be to find other and better processes and tools to rely on. And then actually use them.

I think that is part of what I was attempting to get at, though I probably didn't do a very good job. In a sense we are biased to use certain processes or tools. The only way to change those "default settings" is to deliberately practice something better, so that when the time comes, you'll be ready.

You can't go wrong with writing, as it is nearly universally required and will be among the last skills to be Turing'd. A STEM major who can write well is a scarce commodity, if the GRE scores are any indication. (I would not recommend taking a course to improve your writing however. Just start writing a blog or something. And read Eats, Shoots & Leaves.)

If you have trouble socializing, learn how to do that. Most jobs opportunities come through socializing. Even the jobs that are advertised still require a competent interview.

"The subject is confronted with the evidence that his wife is also his mother, and additionally with the fact that this GLUT predicts he will do X". Is it clear that an accurate X exists?

You mean, he is confronted with the statement that this GLUT predicts he will do X. That statement may or may not be true, depending on his behavior. He can choose a strategy of either always choosing to do what is predicted, always choosing to do the opposite of what is predicted, or ignoring the prediction and choosing based on unrelated criteria. It is possible to construct a lookup table containing accurate predictions of this sort only in the first and third cases, but not in the second.

It certainly doesn't represent mine. The architectural shortcomings of narrow AI do not lend themselves to gradual improvement. At some point, you're hamstrung by your inability to solve certain crucial mathematical issues.

Narrow-AI driverless cars will probably not decide that they need to take over the world in order to get to their destination in the most efficient way. Even if it would be better, I would be very surprised if they decided to model the world that generally for the purposes of driving.

There's only so much modeling of the world/general capability you need in order to solve very domain-specific problems.