Introspection tells me this statement usually gets trotted out when the cost of achieving fairness is too high to warrant serious consideration.

Could you give an example of such a situation where the cost of achieving "fairness" is indeed too high for you? Because I have a hunch that we differ not so much in our assessment of costs but in our notions of "fairness". Oh, and what is "Serious consideration"? Is a young man thinking of what route he should set his life upon and wanting to increase "fairness" doing more or less serious consideration than an adult thinking whether to give $500 to charity?