If you're a human and you want to have correct beliefs, you must make a special effort to seek evidence that your beliefs are wrong. One of our known defects is our tendency to stick with our beliefs for too long. But if you do this successfully, you will become less certain and therefore less determined.

Normatively, seeking disconfirmation and not finding it should make you more certain. And if you do become less certain, I'm not convinced this necessarily makes you less determined – why couldn't it heighten your curiosity, or (especially if you have something to protect) make you more determined to try harder and return, with justification, to the same certainty?

Who do you think is going to be more motivated to think about math: someone who feels it is their duty to become smarter, or a naive student who believes he or she has the answer to some mathematical problem and is only lacking a proof?

Or, how about the student who believes they may have the answer, and has a burning itch to know whether this is the case? Or the one with something to protect?

You rarely see a self-help book, entreprenuership guide, or personal development blog telling people how to be less confident.

While I'm not very familiar with these literatures, I suspect encouraged overconfidence is often just a motivational hack, in which case you should again look for a Third Alternative: can you find the willpower to do this, while having properly calibrated belief in its success? Alternately, it might hack around people not realizing (even deliberatively) the potential payoff of success and/or the upside of failure; but then you should determine and appreciate these things.

Also, y'know, there could be some issues on which many people really are underconfident.