I suspect the "getting it" versus "not getting it" is largely derived from different baselines. If you choose a social-normative baseline, "privilege" makes no sense, because what you're really talking about is a lack of handicaps, and the implication that somebody is better off because they -aren't- handicapped, rather than because of their own efforts (which somehow always gets suggested in privilege discussions), is offensive. If you choose a lower baseline, "handicap" is just offensive, because really, people who are better off than the baseline are enjoying a special status they refuse to acknowledge.

Personally, I find it a gross misuse of the word "privilege" to begin with, and a blatant attempt at connotation mining a word which had its connotative roots in its meaning of special legal rights enjoyed by a legally superior class. Its use in a non-legal context, to describe the absence of social or physical handicaps, was initiated in a deliberate attempt at mental subversion, to bypass rationality with the emotive appeal of a word. But then, I ascribe to a higher baseline of "normal."