The "genetic lottery" as you put it works on specific traits;

Yes, but the selection works from two directions: to be an Olympic-class athlete you need to a have a major advantage in some particular trait and no noticeable fitness disadvantages. In a random selection from the population there will be somewhat-ill people (e.g. with chronic diseases, say, autoimmune), but these people will be absent from the Olympic athlete sample. That by itself is probably enough to generate a noticeable longevity advantage for the athletes.