Is there any way to do these things without paying a large pricetag? Could you just lurk around campus or something? Only half-joking here.

be sure to first consider the most useful version of grad that you could reliably make for yourself... and then decide whether or not to do it.

Planning fallacy is going to eat you alive if you use this technique.

Predictably, Naruto turns out to have inherited all his abilities from his parents, and then improved on them only because he was possessed by the ancient spirit of one of the most powerful beings in existence. And even before that, when the story required him to be the underdog hero, he tended to overcome obstacles using the Kyuubi.

All of the Narutoverse in general is about magic powers (chakra, whatever) passing on from parents to children without much of a change. There's exactly one character in all of Narutoverse who's called out for being powerful due to training, and it isn't Naruto. (A few others are powerful due to research, which is of course always evil.)

Naruto is the opposite of Tsuioku Naritai. It's the story of "everyone had something to protect and practiced like mad, but none of it made a huge difference and most everyone would have been about as powerful anyway." Naruto climbs trees (metaphorically speaking) for many chapters, but keeps being the underdog. Then he starts manifesting powers that make him the most powerful individual in the universe - because he's a shonen hero - and which are entirely due to his parents and outside intervention.

Naruto. Lots of wanting to be stronger, and training hard in order to become stronger, often as a response to frustration about not being strong enough yet. Tree climbing example.

There are probably many other examples of Japanese fiction with similar themes (Eliezer basically said as much), but Naruto is the one that I'm familiar with.

That is why I looked at all 10 questions in aggregate.

Well, you did not look at calibration, you looked at overconfidence which I don't think is a terribly useful metric -- it ignores the actual calibration (the match between the confidence and the answer) and just smushes everything into two averages.

It reminds me of an old joke about a guy who went hunting with his friend the statistician. They found a deer, the hunter aimed, fired -- and missed. The bullet went six feet to the left of the deer. Amazingly, the deer ignored the shot, so the hunter aimed again, fired, and this time the bullet went six feet to the right of the deer. "You got him, you got him!" yelled the statistician...

So, no, I don't think that overconfidence is a useful metric when we're talking about calibration.

but I also did another analysis which looked at slopes across the range of subjective probabilities

Sorry, ordinary least-squares regression is the wrong tool to use when your response variable is binary. Your slopes are not valid. You need to use logistic regression.

D_Malik's scenario illustrates that it doesn't make sense to partition the questions based on observed difficulty and then measure calibration, because this will induce a selection effect. The correct procedure to partition the questions based on expected difficulty and then measure calibration.

Yes, I agree with that. However it still seems to me that the example with coins is misleading and that the given example of "perfect calibration" is anything but. Let me try to explain.

Since we're talking about calibration, let's not use coin flips but use calibration questions.

Alice gets 100 calibration questions. To each one she provides an answer plus her confidence in her answer expressed as a percentage.

In both yours and D_Malik's example the confidence given is the same for all questions. Let's say it is 80%. That is an important part: Alice gives her confidence for each question as 80%. This means that for her the difficulty of each question is the same -- she cannot distinguish between then on the basis of difficulty.

Let's say the correctness of the answer is binary -- it's either correct or not. It is quite obvious that if we collect all Alice's correct answers in one pile and all her incorrect answers in another pile, she will look to be miscalibrated, both underconfident (for the correct pile) and overconfident (for the incorrect pile).

But now we have the issue that some questions are "easy" and some are "hard". My understanding of these terms is that the test-giver, knowing Alice, can forecast which questions she'll be able to mostly answer correctly (those are the easy ones) and which questions she will not be able to mostly answer correctly (those are the hard ones). If this is so (and assuming the test-giver is right about Alice which is testable by looking at the proportions of easy and hard questions in the correct and incorrect piles), then Alice fails calibration because she cannot distinguish easy and hard questions.

You are suggesting, however, that there is an alternate definition of "easy" and "hard" which is the post-factum assignment of the "easy" label to all questions in the correct pile and of the "hard" label to all questions in the incorrect pile. That makes no sense to me as being an obviously a stupid thing to do, but it may be that the original post argued exactly against this kind of stupidity.

P.S. And, by the way, the original comment which started this subthread quoted Yvain and then D_Malik pronounced Yvain's conclusions suspicious. But Yvain did not condition on the outcomes (correct/incorrect answers), he conditioned on confidence! It's a perfectly valid exercise to create a subset of questions where someone declared, say, 50% confidence, and then see if the proportion of correct answers is around that 50%.

There's been far less writings on improving rationality here on LW during the last few years. Has everything important been said about the subject, or have you just given up on trying to improve your rationality? Are there diminishing returns on improving rationality? Is it related to the fact that it's very hard to get rid off most of cognitive bias, no matter how hard you try to focus on them? Or have people moved talking about these on different forums, or in real life?

Or like Yvain said on 2014 Survey results.

It looks to me like everyone was horrendously underconfident on all the easy questions, and horrendously overconfident on all the hard questions. To give an example of how horrendous, people who were 50% sure of their answers to question 10 got it right only 13% of the time; people who were 100% sure only got it right 44% of the time. Obviously those numbers should be 50% and 100% respectively.

This builds upon results from previous surveys in which your calibration was also horrible. This is not a human universal - people who put even a small amount of training into calibration can become very well calibrated very quickly. This is a sign that most Less Wrongers continue to neglect the very basics of rationality and are incapable of judging how much evidence they have on a given issue. Veterans of the site do no better than newbies on this measure.

You're safeguarding against the wrong thing. If I needed to fake a prediction that badly, I'd find a security hole in Less Wrong with which to edit all your comments. I wouldn't waste time establishing karma for sockpuppets to post editable hashes to deter others from posting hashes themselves, that would be silly. But as it happens, I'm not planning to edit this hash, and doing that wouldn't have been a viable strategy in the first place.

Nice post!

Related: Being proven wrong is like winning the lottery by Phil Birnbaum, How to enjoy being wrong by lincolnquirk

I'm about to start being paid for a job, and I was looking at investment advice from LW. I found this thread from a while back and it seemed good, but it's also 4 years old. Can anyone confirm if the first bullet is still accurate? (get VTSMX or VFINX on vanguard, it doesn't matter too much which one.)

Suppose you're Bayesian, and you're calculating

P(lead causes crime | data) = P(data | lead causes crime) * P(lead causes crime) / P(data).

What Pinker is saying is that P(data | lead causes crime) is not as high as you'd think, because if lead really does cause crime, we should not expect the crime curve to be a time-shifted version of the lead curve. It's probably still true that P(data | lead causes crime) > P(data), so that you should update in the direction of lead causes crime, but this update should probably be smaller than you thought before reading that paragraph.