If smart people are saying "2+2=5" and I point out it's 4, would you say "what matters is why you want to know what 2+2 is"?

Yes. For example, let's take a clearer mathematical statement, "3 is prime". It seems that's true whatever people say. However, if you come across some mathematicians who are having a discussion that assumes 3 is not prime, then you should think you're missing some context rather than that they are bad at math.

I chose this example because I once constructed an integer-like system based on half-steps (the successor function adds .5). The system has a notion of primality, and 3 is not prime.