doesn't come across to me as arrogant, hostile, or assuming any kind of relationship of superiority in the first place.

Hostile, no. Arrogant -- a bit, but quite within the LW norm. But asserting superiority? Very much so. Here is a direct quote:

I outstrip all but a small handful of LWers in intellectual caliber by a very large margin.

And the problems arose not because of claims about superior domain knowledge, but rather claims about superior "crystallized intelligence" and "intellectual caliber" which are much wider than "I'm really good at math".