When I say that "2 + 2 = 4" is a fact independent of the physical universe, I mean that it is the sort of information that does not tell us (directly) which universe we are in, what the laws of physics are. (Though it can be used in a line of reasoning evaluating propositions that are about our particular universe.) I do not mean that it is a supernatural force, or a mystical energy field controlling our destinies.

And the whole purpose of making that point was to frame the question of how well the physical systems in our brains represent these logical facts, and how we can evaluate our uncertainty about the answer to that question.