Confidence is a state of mind. It is critical from the standpoint of motivation. Without confidence we would be paralyzed into inaction; we would be unable to turn decisions into structured consequences. We would be constantly "scoping the game plan" and never playing. However, confidence should not play a major role in making decisions. Cold rationality is key in two aspects of the decision process: (a) how important is the decision? (b) if the decision is important, what is the "outside view" ? (per Kahneman) The first decision, IMO, should be handled with an algorithm analogous to triage and requires ascertaining sufficient basic facts to determine what, in fact, is the decision that needs to be made and how long can the decision be deferred. In other words, part of the algorithm might be answering the question, 'what happens if we do nothing?' If the decision appears essentially trivial (i.e., should I buy a new chair and, if yes, should I buy the red chair or the blue), you don't need to get to (b). If a decision is important, you need to use cold rationality.
If I am dealing with a situation where the decision has already been made, I may be able to use learned skills and experience to determine how to act. Then the key question from the standpoint of confidence is whether the situation falls within the scope of my expertise, where I can be confident that my trained 'gut reaction' will be an appropriate response. If it is outside my area of expertise, I have no reason to be confident - although I may act like I am confident if success depends upon others trusting my abilities.
Regardless of how I may need to appear to others, I would never try to kid myself about my abilities. What may be missing in the above question - Should I believe hard that I can accomplish X regardless of the likelihood of success - are the foundational questions: Do I really have to try to accomplish X? Is there a reasonable alternative method that is more likely to be successful? Is there a reasonable alternative outcome Y that will give me the benefits I need from X with a greater chance of success. If the answers are Yes, No, No - then you have to believe in order to win, so throw the "Hail Mary" with total confidence.
What should a rationalist do about confidence? Should he lean harder towards
I don't want to falsely construe these as dichotomous. The real answer will probably dissolve 'confidence' into smaller parts and indicate which parts go where. So which parts of 'confidence' correctly belong in our models of the world (which must never be corrupted) or our motivational systems (which we may cut apart and put together however helps us achieve our goals)? Note that this follows the distinction between epistemic and instrumental rationality.
Eliezer offers a decision criterion in The Sin of Underconfidence:
It makes us stronger to know when to lose hope already, and it makes us stronger to have the mental fortitude to kick our asses into shape so we can do the impossible. Lukeprog prescribes boosting optimism "by watching inspirational movies, reading inspirational biographies, and listening to motivational speakers." That probably makes you stronger too.
But I don't know what to do about saying 'I can do it' when the odds are against me. What do you do when you probably won't succeed, but believing that Heaven's army is at your back would increase your chances?
My default answer has always been to maximize confidence, but I acted this way long before I discovered rationality, and I've probably generated confidence for bad reasons as often as I have for good reasons. I'd like to have an answer that prescribes the right action, all of the time. I want know when confidence steers me wrong, and know when to stop increasing my confidence. I want the real answer, not the historically-generated heuristic.
I can't help but feeling like I'm missing something basic here. What do you think?