If fairness is about something other than human agreement, what is it?

Suppose you have a rule that you say is always the fair one. And suppose that you apply it to a situation involving N people, and all N of them object, none of them think it's fair. Are you going to claim that the fair thing for them to do is something that none of them agrees to? What's fair about that?

When everybody involved in a deal agrees it's fair, who are you -- an outside kibitzer -- to tell them they're wrong?

Suppose a group all agrees, they think a deal is fair. And then you come in and persuade some of them that it isn't fair after all, that they should get more, and the agreement breaks down. Maybe they fight each other over it. Maybe some of them get hurt. And after some time contending, it's clear that none of them are better off than they were when they had their old agreement. Were you being fair to that group by destroying their agreement?