This seems like an unproductive attitude. If everyone took this attitude they would never have hammered out the problems in calculus in the 19th century. And physicists would probably not have every discovered renormalization in the 20th century.
I will make sure to not spread that opinion in the event that I travel back in time.
A better approach is to try to define rigorously what is happening with the infinities. When you try that, either it becomes impossible (that is there's something that seems intuitively definable that isn't definable) or one approach turns out to be correct, or you discover a hidden ambiguity in the problem. In any of those cases one learns a lot more than simply saying that there's an infinity so one can ignore the problem.
I agree with you that this is a better approach. However, the problem in question is "find the error" not "how conservation of momentum works," and so as soon as you realize "hm, they're not treating this infinity as a limit" then the error is found, the problem is solved, and your curiosity should have annihilated itself.
We've discussed Edward Nelson's beliefs and work before. Now, he claims to have a proof of a contradiction in Peano Arithmetic; which if correct is not that specific to PA but imports itself into much weaker systems. I'm skeptical of the proof but haven't had the time to look at it in detail. There seem to be two possible weakpoints in his approach. His approach is to construct a system Q_0^* which looks almost but not quite a fragment of PA and then show that PA both proves this system's consistency and proves its inconsistency.
First, he may be mis-applying the Hilbert-Ackermann theorem-when it applies is highly technical and can be subtle. I don't know enough to comment on that in detail. The second issue is that in trying to show that he can use finitary methods to show there's a contradiction in Q_0^* he may have proven something closer to Q_0^* being omega-inconsistent. Right now, I'm extremely skeptical of this result.
If anyone is going to find an actual contradiction in PA or ZFC it would probably be Nelson. There some clearly interesting material here such as using a formalization of the surprise examiation/unexpected hanging to get a new proof of of Godel's Second Incompleteness Theorem. The exact conditions which this version of Godel's theorem applies may be different from the conditions under which the standard theorem can be proven.