The fact that someone uses a calibration system different than yours does not make theirs "poorer" than yours.

It does, in a certain useful sense. The truth of how well a belief works is not directly accessible, so all you have is an estimate. If, after taking into account the fact that another person's belief is different, your beliefs remain different, you should expect their belief to be worse than yours (otherwise, you should just accept their belief). They should sometimes estimate that their belief is better than yours as well, at the same time. In this situation, one of you is wrong about the estimate of whose belief is more accurate, but this is similar to how one of you has a more accurate belief than the other, that is one of you is more wrong about the fact in question that the belief estimates.

Otherwise, what's to stop me from saying that "if you think it isn't rude then you're the one who's poorly calibrated"?

Suppose nothing does; even in that case it is not a relevant consideration. (In practice, it is the fact that it's already known at this point in the conversation that the argument won't be accepted by the interlocutor without additional justification.)