System 2 is not enough. But if there is a straw of hope to mitigate the bias in a particular situation where finding a correct solution is important, the only chance to find it is to search for it using your System 2.

No, you can also take a good night sleep to give your System I more time to anaylse the situation to improve it's output.

You can also do exercises to calibrate your credence. Calibration training for probability estimates is probably one of the best ways to get them right.