You're over-thinking this. Take a look at this real-world example of a "neurological fault":

Now I knew where I was. Soon I would come to interchange 27 with its two ramps, A and B. B led away from my destination and A directly into it. It had always struck me as strange that one reached 27B before 27A. I recalled drawing that on a map to give to someone who was going to visit me. My breathing has returned to normal and my panic had disappeared. I come up to the first sign for the interchange.

"27A"

I could hardly breathe. That was not possible. 27A was after 27B. I knew that. I considered for a moment the possibility that on the previous night, shortly after I drove on this very highway, construction workers had descended en masse on the interchanges and somehow moved them. That seemed far more possible than that my clear (and detailed) memory could be so wrong. 27A looked exactly as I remembered, except that now I could see 27B clearly in the distance and in the past I had to turn my head to see it.

I exited on the ramp that I knew wasn’t there twenty-four hours previously to find myself on a well-remembered road. And soon I was home.

Now imagine that happening on a massive scale. Say that right after reading this comment you experience evidence, like that which the OP describes, going against your memories of what happens when you put one pair of objects next to another pair. (This includes "mental confirmation that XXX - XX = XX", though not a formal proof in PA.) Would that make you doubt your memories of what PA says? Would you want to check the proof that it says (2+2=4) in case your current memory is lying about that as well?