Looking at your comment I am not sure we disagree. Rather than unpacking what I'm trying to get at (which is orthogonal to the discussion), I'll leave it be. But if you are curious look up "heritabitility" in Hartl & Clark, they explain the issues more lucidly than I could here.