Hm. Well, one person's casually obvious fact is another's unverified anecdote.

In my experience, physicists are typically skinny, toned, and in reasonably good cardiovascular shape, but not necessarily "good at sports compared to typical people" after adjusting for education and free time / flexibility of work hours. They tend to be good at running, swimming, pushups, karate, and Ultimate Frisbee, but not necessarily at tennis, football, wrestling, soccer, golf, basketball, or skiing. Again, that's just my biased set of estimates based on a nonrandom sample of about 10 physicists and 20 people with comparable lifestyles but other careers. I wouldn't expect it to be at all representative of national or global trends, and I'm surprised that you feel confident about your assertion.