I still feel confused. I definitely see that, when we talk about fairness, our intended meaning is logical in nature. So, if I claim that it is fair for each person to get an equal share of pie, I'm trying to talk about some set of axioms and facts derived from them. Trying.

The problem is, I'm not convinced that the underlying cognitive algorithms are stable enough for those axioms to be useful. Imagine, for example, a two-year-old with the usual attention span. What they consider "good" might vary quite quickly. What I consider "just" probably depends on how recently I ate. Even beyond such simple time dependence, what I consider "just" will definitely depend on context, framing, and how you measure my opinion (just ask a question? Show me a short film? Simulate the experience and see if I intervene?). Part of why friendly AI is so hard is that humans aren't just complicated, we're not even consistent. How, then, can we axiomatize a real human's idea of "justice" in a useful way?