Hm. Well, one person's casually obvious fact is another's unverified anecdote.
In my experience, physicists are typically skinny, toned, and in reasonably good cardiovascular shape, but not necessarily "good at sports compared to typical people" after adjusting for education and free time / flexibility of work hours. They tend to be good at running, swimming, pushups, karate, and Ultimate Frisbee, but not necessarily at tennis, football, wrestling, soccer, golf, basketball, or skiing. Again, that's just my biased set of estimates based on a nonrandom sample of about 10 physicists and 20 people with comparable lifestyles but other careers. I wouldn't expect it to be at all representative of national or global trends, and I'm surprised that you feel confident about your assertion.
That sounds right on all points. My guess is that you are just using an unrealistic baseline. Typical people really are shockingly bad at everything.
tl;dr: The marginal benefits of learning a skill shouldn't be judged heavily on the performance of people who have had it for a long time. People are unfortunately susceptible to these poor judgments via the representativeness heuristic.
Warn and beware of the following kludgy argument, which I hear often and have to dispel or refine:
"Naively, learning «skill type» should help my performance in «domain». But people with «skill type» aren't significantly better at «domain», so learning it is unlikely to help me."
In the presence or absence of obvious mediating factors, skills otherwise judged as "inapplicable" might instead present low hanging fruit for improvement. But people too often toss them away using biased heuristics to continue being lazy and mentally stagnant. Here are some parallel examples to give the general idea (these are just illustrative, and might be wrong):
Aside from easily identifiable particular flaws [as SarahC points out, the difference between an athelete and a physicist isn't just physical activity], there are a few generic reasons why these arguments are weak:
All this should be taken into account before dismissing the new skill option. In general, try to flesh out the analysis with the following themes:
So yeah, don't let specialists over-represent the skills the specialize in. Many readers here are in the "already have it" category for a lot of the skills I'm talking about, and there are already lots of posts convincing us to decompartmentalize those skills… but it's also helpful to consider the above ideas in balance with the legitimate counterarguments when convincing others to learn and apply new skills.