it seems like very few people I hear using the term that way realize that a madrassa is often explicitly secular.

That's not totally accurate. While in some dialects of Arabic 'madrassa' simply means 'school' in both secular and religious settings, in other dialects the word refers only to where religious Islamic teaching takes place, and not secular schools.