I've been considering the hypothesis that mathematical intuition (especially intuition about highly abstract, non-physical things) comes from an ability to model that math physically in the brain. When we interrogate our 'intuition', we're actually interrogating these (mechanical) models.

This is correct, but there is a useful layer of abstraction to consider. There are a set of operations the brain does that we can be conscious of doing, and inspect the structure of how they interact within our own brains. These operations are, of course, implemented by physics, they come from the structure of neurons and other supporting biological components. And therefore, the structures that are built out of these operations are also, ultimately, implemented by physics, though a lot can be learned by looking at the introspectively observable structure. These operations can be used to build a model of arithmetic. This does give us some power to "explain in detail how we knew".