The "controversy" about infinite sets is about their existence/usefulness as physical objects, not their mathematical existence (you'll note that I was careful to say that they were not physical objects in the grandparent). From that article:
"Infinite set atheism" is a tongue-in-cheek phrase used by Eliezer Yudkowsky to describe his doubt that infinite sets of things exist in the physical universe.
Thus, infinite sets are a perfect example of a mathematical object disconnected from physical reality/practical experience.
We can construct the natural numbers by starting with two symbols "0" and "1" that are naturals, and saying that if n is a natural, then n+1 is too i.e. adding 1 over and over again. Part of the definition is each time we add 1, we get a number we haven't seen before; and so we have an infinite set by construction. And we can make bigger ones by taking the power set (the power set always has a larger cardinality then the set it comes from).
So infinite sets are definitely mathematical objects because we can (and just have) construct them.
Watch Eliezers response to this question, http://www.youtube.com/watch?v=3dufqGC8X8c
scroll to 4:40 I like his one argument: if we have finite neurons and thus cannot construct an infinite set in our "map" what makes you think that you can make it correspond to a (hypothetical) infinity in the territory?
Here's the new thread for posting quotes, with the usual rules: