This wouldn't be evidence that 2+2=3 in Peano arithmetic, it would be evidence that Peano arithmetic just doesn't apply for some reason.

Exactly.

My point is that 2+2=4 in Peano Arithmetic independent of observation, but observation tells you if Peano arithmetic applies or not.

It is worth emphasizing that to claim that "2+2=4 in Peano Arithmetic independent of observation" is not to claim that our knowledge of this fact about Peano Arithmetic is independent of observation. (The former claim is about our map of the territory; the latter is about our map of our map of the territory.)