If you are told a billion dollars hasn't been taxed from people in a city, how many people getting to keep a thousand dollars (say) do you imagine? Probably not a million of them. How many hours not worked, or small things that they buy do you imagine? Probably not any.

But now that I think about it, I'd rather have an extra thousand dollars than be able to drink at a particular drinking fountain.

But I don't think fairness the morality center is necessarily fairness over differing amounts of harm. It could be differing over social status. You could have an inflated sense of fairness, so that you cared much more than the underlying difference in what people get.