That said: I'm under no obligation (moral, rational, or otherwise) to respect or value Sam in this way.

I actually agree with you on this. There are times when I assign a high probability on an argument being fruitless and so don't engage in the argument. My point was not that one must always argue, only that when one chooses not to argue, it is, as you say, evidence that they respect and value the person less. Even then, I would say that the decision not to argue should have some small amount of pain to it: if nothing else, sadness at the realization that the other person isn't worth your time. (I believe this will help with boundary cases where we are in agreement.)

To that end, it is notable that Swimmer is not the one who chose to not argue. Charlie chose that. Which is evidence that Charlie does not value criticism and is evidence for his lack of respect for the position that he is wrong. And that realization should cause one to gnash one's teeth a little if one is put into a position where Charlie is allowed to say his piece and Swimmer is not.