If you can make a chord, you are counting the distance from the other keys.

This appears to me to be a highly nonstandard use of the word "counting".

Nobody can count to 12 a thousand times a minute and be terrible at math.

They most certainly can. Counting to 12 is a teeny tiny weeny itsy bitsy fraction of mathematics. You can be very good at counting to 12 and terrible at mathematics overall.

The issue here is you think arithmetic isn't math.

It seems more as if the issue is that you think arithmetic is all there is to mathematics. I'm sure you don't actually think that, because it's silly, but I don't see how to make any sense of what you're saying without an assumption along those lines.