Your theory doesn't seem to stand up to the data, here and here. It seems it's women that underestimate male attractiveness. Men's judgments are almost symmetrical. Data is from Okcupid surveys.

(obvious confounds: people that use okcupid may not be representative of the population generally, both for the raters and ratees.)