Yes, 'fairness' is often a concept that gets invoked in these sorts of discussions.

For my own part, given world W1 where I have X1 and the best-off people have Y1, and world W2 where I have X2 and the best-off people have Y2, such that (X2 < X1) and (Y2-X2) << (Y1 - X1), within a range of worlds such that X1 and X2 are both not vastly different from what I have today, I expect that when transitioning from W2 to W1 I would experience myself as better off, and when transitioning from W1 to W2 I would experience myself as worse off.

I expect that's true of most people.

It's not necessarily the only important question here, though.