But there's certainly no evidence supporting the idea that this is likely to happen, nor is the anxiety helping me prevent it or helping me in any other way.

A simple way to put that anxiety to use would be whenever you feel that anxiety, to do 10 pushups. Becoming more fit makes you more attractive, so it's not irrational to start doing more sport to increase your chances with woman.

Channel the anxiety into some useful activity.