I don't think it has much to do with personality either, except, like you said, willingness to work hard (especially if you're someone who starts out finding it very difficult.) But I think that a lot of people, even people who can work up to the level of calculus in math, go at it with the mindset of "memorize that Formula X gives Answer Y" instead of trying to understand how and why Formula X relates to the underlying structure of the problem so that it's obvious that it should give answer Y, but gives Answer Z in a different context... You can get by with memorizing formulas in math classes, at least the way they're currently taught and tested. It's a lot harder to get by with that habit that when you're programming.

(On the whole, the people I've known whose minds appear to work like this aren't noticeably "lower" intelligence, however you define that. They just don't think of math as something where they should be applying the analytic part of their mind.)