That can happen sometimes but it can also go the other way. Testosterone has its downsides. Its upsides might still outweigh them, obviously.
I was born with high prenatal testosterone, so I can attract people now even without testosterone. I generally go without testosterone, 'cuz I think it negatively affects my rationality. Hypothetically, this restricts me to cardio, and keeps me from lifting. In practice, though, it doesn't really come up.
Testosterone has its downsides.
Messes with cholestrol levels something shocking. Makes you stinky. Causes acne. Is converted to DHT leading to male pattern baldness and
You actively avoid lifting weights so as to reduce your testosterone levels? That's not something I've heard before.
An interesting blog post by Razib Khan, who many here probably know from his Gene Expression blog, the old gnxp site or perhaps from his BHTV debate with Eliezer.
I recommend following the link and reading the rest of it there, not only does interestingness continue, the comment section there is usually worth reading since he vigorously moderates it.