From the inside it can be very tough to tell, but from the outside they're clearly they're wrong about them all being low probability.

I don't know about that. That clearly depends on the situation -- and while you probably have something in mind where this is true, I am not sure this is true in the general case. I am also not sure of how would you recognize this type of situation without going circular or starting to mumble about Scotsmen.

if you learn you're wrong, if you can't learn how you're wrong and in which direction to update even after thinking about it

What do you mean, can you give some examples? Normally, if people locked themselves in a box of their own making, they can learn that the box is not really there.

The idea of making mistakes, not realizing, and then using that lack of realization as further evidence that I'm not overconfident is a trap I don't want to fall into.

That's a good point -- I agree that if you don't realize what opportunity costs you are incurring, your cost-benefit analysis might be wildly out of whack. But again, the issue is how do you reliably distinguish ex ante where you need to examine things very carefully and where you do not have to do this. I expect this distinguishing to be difficult.

"Actually thinking it through" is all well and good, but it basically boils down to "don't be stupid" and while that's excellent advice, it's not terribly specific. And "can you eat the loss?" is not helping much. For example, let's say one option is me going to China and doing a start-up there. My "internal model" says this is a stupid idea and I will fail badly. But the "loss" is not becoming a multimillionaire -- can I eat that? Well, on the one hand I can, of course, otherwise I wouldn't have a choice. On the other hand, would I be comfortable not becoming a multimillionaire? Um, let's say I would much prefer to become one :-) So should I spend sleepless nights contemplating moving to China?