Well I agree that I can think just with verbal signs, so long as the verbal sentences or symbolic statements mean something to me (could potentially pay rent*) or the symbols are eventually converted into some other representation that means something to me.
I can think with the infinity symbol, which doesn't mean anything to me (unless it means what I first said above: in short, "way big enough"), and then later convert the result back into symbols that do mean something to me. So I'm fine with using infinity in math, as long as it's just a formalism (a symbol) like that.
But here is one reason why I want to object to the "realist" interpretation of infinity via this argument that it's just a formalism and has no physical or experiential interpretation, besides "way big enough": The Christian god, for example, is supposed to be infinite this and infinite that. This isn't intended - AFAIK - as a formalism nor as an approximation ("way powerful enough"), but as an actual statement. Once you realize this really isn't communicating anything, theological noncognitivism is a snap: the entity in question is shown to be a mere symbol, if anything. (Or, to be completely fair, God could just be a really powerful, really smart dude.) I know there are other major problems with theology, but this approach seems cleanest.
*ETA: This needs an example. Say I have a verbal belief or get trusted verbal data, like a close friend says in a serious and urgent voice, "(You'd better) duck!" The sentence means something to me directly: it means I'll be better off taking a certain action. That pays rent because I don't get hit in the head by a snowball or something. To make it into thinking in words (just transforming sentences around using my knowledge of English grammar), my friend might have been a prankster and told me something of the form, "If not A, then not B. If C, then B. If A, then you'd better duck. By the way, C." Then I'd have to do the semantic transforms to derive the conclusion: "(I'd better) duck!" which means something to me.
To know reality we employ physics. Physics employs calculus. Calculus employs limits. Limits employ infinite sequences. Does that pay enough rent?
[edit: sorry, the formatting of links and italics in this is all screwy. I've tried editing both the rich-text and the HTML and either way it looks ok while i'm editing it but the formatted terms either come out with no surrounding spaces or two surrounding spaces]
In the latest Rationality Quotes thread, CronoDAS quoted Paul Graham: