I am and your understanding is correct for most applications. I don't think it matters for this question, as my understanding is that the operative factor behind the Aharonov-Bohm effect is the nonlocality of wavefunctions.* Because wavefunctions are nonlocal, the potential formulation is staggeringly simpler than a force formulation. (The potentials are more real in the sense that the only people who do calculations with forces are imaginary!)

You still have gauge freedom with the Aharonov-Bohm effect- if you adjust the four-potential everywhere, all it does is adjust the phase everywhere, and all you can measure are phase differences.

Although, that highlights an inconsistency: if I'm willing to accept wavefunctions as real, despite their phase freedom, then I should be willing to accept potentials are real, despite their gauge freedom. I'm going to think this one over, but barring any further thoughts it looks like that's enough to change my mind.

*I could be wrong: I have enough physics training to speculate on these issues, but not to conclude.

[edit] It also helps that Feynman, who certainly knows more about this than I do, sees the potentials as more real (I suppose this means 'fundamental'?) than the fields.