I think this only works if you keep 'mathematics' a broad and vague category.

I agree. But group theory isn't less "math" than actuarial math is!! I'd be happy with just dropping "math" as a term and renaming second semester calculus "computing integrals" or something. Then Scott could say that he's bad at computing integrals, rather than thinking that because he's bad at computing integrals, group theory must be way beyond him.

But since people call both calculus class and group theory "math," I need to respond to that.

So, I'll defer to Scott if he disagrees, but my impression is that he has substantially more trouble learning scales and chords than his brother, and that he is "worse than him at music." It might not be logically necessary, but we can certainly notice that it is probabilistically likely.

Yes, but I don't think that Scott's innately worse at music than most people who can easily pick up on scales and chords, the countervailing forces cutting in his favor are too strong.