1) I think your reaction to this situation owed itself more to your psychological peculiarities as a person (whichever they are) than to a characteristic that all people that identify as rationalists share. There's no reason to expect people with the same beliefs as yours never to keep their cool (at least never on the first time) when talking to someone with an obviously incompatible belief system.
2)
It was extremely difficult to construct an argument against, because all of my arguments had logically consistent bases, and were thus invalid in its face.
It doesn't have to be like that, at least if you don't start off with consistent and false belief systems. The way I think about such issues while effectively avoiding epistemological crises is the following: an algorithm by which I arrived at conclusions which I can pretty confidently dub "knowledge" ends up being added to my cognitive toolbox. There are many things out there that look like they were designed to be added to people's cognitive toolboxes, but not all of them can be useful, can they? Some of them look like they were specifically designed to smash other such tools to pieces. So here's a good rule of thumb: don't add anything to your cognitive toolbox that looks like an "anti-tool" to a tool that is already inside of it. Anything that you suspect makes you know less, be dumber, or require you to forsake trustworthy tools is safe & recommendable to ignore. (In keeping with the social justice topic, a subcategory of bad beliefs to incorporate are those that cause you to succumb to, rather than resist, what you know to be flaws in your cognitive hardware, such as an ingroup-outgroup bias or affect heuristics -- that's why, I think, one should avoid getting too deep into the "privilege" crowd of social justice even if the arguments make sense to one.) Of course, you should periodically empty out the toolbox and see whether the tools are in a good state, or if there's an upgraded version available, or if you were simply using the wrong hammer all along -- but generally, rely on them.
3) You like to explore the implications of a premise, which is completely incompatible with your friend's "separate magisteria" approach (a technique directly out of the Official Handbook of Belief Conservation); unfortunately it is why you weren't able to abandon the train of thought before it derailed into emotional disturbance. You see someone saying you shouldn't use an obviously (to you) useful and relevant method for investigating something? That's a sign that says "Stop right here, there's no use in trying to extrapolate the consequences of this belief of theirs; they obviously haven't thought about it in sufficient detail to form opinions on it that you can make head or tails of." The knowledge and deepness of thought that it takes to see why math is relevant to understanding society is small enough that, if they failed to catch even that, they obviously went no further in establishing beliefs about math that could be either consistent or inconsistent with the pursuit of justice and equality. You went as far as seeing the implications and being horrified -- "How can anyone even think that?" -- but it is a thought they likely didn't get to think; the ramifications of their thought about math ended long before that, presumably at the point when it began to interfere with ideological belief conservation.
4) Get better friends. I know the type, and I've learned the hard way not to try to reason with them. Remember that one about playing chess with a pidgeon?
...So here's a good rule of thumb: don't add anything to your cognitive toolbox that looks like an "anti-tool" to a tool that is already inside of it. Anything that you suspect makes you know less, be dumber, or require you to forsake trustworthy tools is safe & recommendable to ignore. (In keeping with the social justice topic, a subcategory of bad beliefs to incorporate are those that cause you to succumb to, rather than resist, what you know to be flaws in your cognitive hardware, such as an ingroup-outgroup bias or affect heuristics -- that
Usually, I don't get offended at things that people say to me, because I can see at what points in their argument we differ, and what sort of counterargument I could make to that. I can't get mad at people for having beliefs I think are wrong, since I myself regularly have beliefs that I later realize were wrong. I can't get mad at the idea, either, since either it's a thing that's right, or wrong, and if it's wrong, I have the power to say why. And if it turns out I'm wrong, so be it, I'll adopt new, right beliefs. And so I never got offended about anything.
Until one day.
One day, I encountered a belief that should have been easy to refute. Or, rather, easy to dissect, and see whether there was anything wrong with it, and if there was, formulate a counterargument. But for seemingly no reason at all, it frustrated me to great, great, lengths. My experience was as follows:
I was asking the opinion of a socially progressive friend on what they feel are the founding axioms of social justice, because I was having trouble thinking of them on my own. (They can be derived from any set of fundamental axioms that govern morality, but I wanted something that you could specifically use to describe who is being oppressed, and why.) They seemed to be having trouble understanding what I was saying, and it was hard to get an opinion out of them. They also got angry at me for dismissing Tumblr as a legitmate source of social justice. But eventually we got to the heart of the matter, and I discovered a basic disconnecf between us: they asked, "Wait, you're seriously applying a math thing to social justice?" And I pondered that for a moment and explained that it isn't restricted to math at all, and an axiom in this context can be any belief that you use to base your beliefs on. However, then the true problem came to light (after a comparison of me to misguided 18th-century philosophes): "Sorry if it offends you, I just don't think in general that you should apply this stuff to society. Like... no."
And that did it. For the rest of the day, I wreaked physical havoc, and emotionally alienated everyone I interacted with. I even seriously contemplated suicide. I wasn't angry at my friend in particular for having said that. For the first time, I was angry at an idea: that belief systems about certain things should not be internally consistent, should not follow logical rules. It was extremely difficult to construct an argument against, because all of my arguments had logically consistent bases, and were thus invalid in its face.
I'm glad that I encountered that belief, though, like all beliefs, since I was able to solve it in the end, and make peace with it. I came to the following conclusions: