An old SO of mine used to get annoyed when I said "quarter of," though.

Was it that phrase in particular? Maybe they're like me (though I don't find it annoying): I've used "half past", "quarter past", and "quarter to" since childhood, but for whatever reason (presumably the phraseology of the people I grew up around) I made it to adulthood not being sure whether "quarter of" meant "quarter to" or "quarter after".

Edit: I know what it means now because I have a friend who uses it. I was born and raised in Michigan, and my friend in New York.