This is something people with only basic exposure to cryptography and security are confused about a lot.
To them "breaking RSA" and "breaking something using RSA" sound essentially the same.
Systems are only as strong as their weakest part.
Nobody bothers attacking basic cryptographic primitives like AES, or RSA, or SHA-1 in practice. These are standardized, simple, have extremely straightforward interfaces, very well researched, and with enormous level of public exposure. Even theoretical weaknesses found occasionally tend to have little practical relevance.
What gets attacked is everything else. Implementations bugs, protocol corner cases, differences between assumptions and actual requirements, side channels, human factors, and very few people looking at it ever. They are highly complicated, messy, with no clear boundaries, and every system starts essentially from scratch.
When P(system is secure) = P(system is secure|AES is secure) P(AES is secure)
You want to focus all your effort on P(system is secure|AES is secure). Both protection effort, and backdooring effort.
RSA is somewhat between the two. It's somewhat more complex, and requirements for securely using RSA are a lot more complicated than for securely using AES. The entire public key crypto is built on far weaker foundations than private key crypto. There's very little hard number theory involved.
When you say "nobody" you mean actual attackers - obviously researchers attack these things all the time. And you do get practical breaks as a result of breaks in cryptography - WEP is the most famous example.
So, the FBI allegedly arranged for a number of backdoors to be built into the OpenBSD IPSEC stack. I don't really know how credible this claim is, but it sparked a discussion in my office about digital security, and encryption in general. One of my colleagues said something to the effect of it only being a matter of time before they found a way to easily break RSA.
It was at about this moment that time stopped.
I responded with something I thought was quite lucid, but there's only so much lay interest that can be held in a sentence that includes the phrases "fact about all integers" and "solvable in polynomial time". The basic thrust of my argument was that it wasn't something he could just decide an answer to, but I don't think he'll be walking away any the more enlightened.
This got me wondering: do arguments that sit on cast-iron facts (or lack thereof) about number theory feel any different when you're making them, compared to arguments that sit on facts about the world you're just extremely confident about?
If I have a discussion with someone about taxation it has no more consequence than a discussion about cryptography, but the tax discussion feels more urgent. Someone walking around with wonky ideas about fiscal policy seems more distressing than someone walking around with wonky ideas about modular arithmetic. Modular arithmetic can look after itself, but fiscal policy is somehow more vulnerable to bad ideas.
Do your arguments feel different?