It seems to me that a) Landsburg's critique is right, b) the problem is still mysterious, c) making it less mysterious would require a new mathematical result, d) that future result is more likely to be negative, like the incompleteness theorem or undefinability of truth.

By far the largest criticism here is the requirements of set theory:

Yudkowsky is relying on a theorem when he says that second-order Peano arithmetic has a unique model. That theorem requires a substantial dose of set theory. So in order to avoid taking numbers as primitive objects, he’s effectively resorted to taking sets as primitive objects. But why is it any more satisfying to take set theory as “given” than to take numbers as “given”? Indeed, the formal study of numbers precedes the formal study of sets by millennia, which suggests that numbers are a more natural starting point than sets are. Whether or not you buy that argument, it’s important to recognize that Yudkowsky has “solved” the problem of accounting for numbers only by reducing it to the problem of accounting for sets — except that he hasn’t even done that, because his reduction relies on pretending that second order logic is logic.

This seems most damning in that it makes most of the philosophical issues about the nature of logic irrelevant.

I think this critique undervalues the insight second order arithmetic gives for understanding the natural numbers.

If what you wanted from logic was a complete, self-contained account of mathematical reasoning then second order arithmetic is a punt: it purports to describe a set of rules for describing the natural numbers, and then in the middle says "Just fill in whatever you want here, and then go on". Landsburg worries that, as a consequence, we haven't gotten anywhere (or, worse, have started behind where we started, since no we need an account of "properties" in order to have an account of natural numbers).

But, thanks to Gödel, we know that a self-contained account of reasoning is impossible. Landsburg's answer is to give up on reasoning: the natural numbers are just there, and we have an intuitive grasp of them. This is, obviously, even more of a punt, and the second order arithmetic approach actually tells us something useful about the concept of the natural numbers.

Suppose someone produces a proof of a theorem about the natural numbers---let's say, for specificity, they prove the twin primes conjecture (that there are infinitely many twin primes). This proof proceeds in the following fashion: by extensive use of the most elaborate abstract math you can think of---large cardinals and sheafs and anabelian geometry and whatnot---the person proves the existence of a set of natural numbers P such that 1) 0 is in P, 2) if x is in P then x+1 is in P, and 3) if x is in P then there are twin primes greater than x.

Second order arithmetic says that this property P must extend all the way through the natural numbers, and therefore must itself be the natural numbers, and so we could conclude the twin primes conjecture.

The rest of this proof is really complicated, and uses things you might not think are valid, so you might not buy it. But second order arithmetic demands that if you believe the rest of the proof, you should also agree that the twin primes conjecture holds. This is actually worth something: it says that if you and I agree about all the fancy abstract stuff, we also have to agree about the natural numbers. And that's the most we can ask for.

To phrase this differently, second order arithmetic gives us a universal way of picking the natural numbers out in any situation. Different models of set theory may end up disagreeing about what's true in the natural numbers, but second order arithmetic is a consistent way of identifying the notion in that model of set theory which lines up with our intuitions for the what the natural numbers are. That's a very different account than the one offered by first order logic, but it's a valuable one in its own right.

One more phrasing, just for thoroughness. Landsburg passes off an account of the natural numbers to Platonism: we just intuit the natural numbers. But suppose you and I both purport to have good intuitions of the natural numbers, but we seem to disagree about some property. So we sit down to hash this out; it might turn out that we have some disagreement about what constitutes valid mathematical reasoning, in which case we may be stuck. But if not---if we agree on the reasoning---then eventually the disagreement will boil down to something like this:

You: "And since 0 is turquoise and, whenever x is turquoise, x+1 is also turquoise, all the natural numbers are turquoise."

Me: "Wait, those aren't all the natural numbers. Those are just the small natural numbers."

You: "What? What are the small natural numbers?"

Me: "You know, the initial segment of the natural numbers closed under the property of being turquoise."

You: "There's an initial segment of the natural numbers which includes 0, is closed under adding 1, but isn't every natural number?"

Me: "Sure. The small natural numbers. Haven't you heard of them?"

You: "Wait, are there any initial segments of the small natural numbers which include 0 and are closed under adding 1?"

Me: "Sure. The tiny natural numbers, for starters."

You: "Look, you're using the word 'natural numbers' wrong. It means..." and then you explain the second order induction axiom. So apparently whatever I thought was the natural numbers was some other bizzarre object; what the second order induction axiom does for a Platonist is pin down which Platonic object we're actually talking about.

Heh - I'm amazed at how many things in this post I alternately strongly agree or strongly disagree with.

It’s important to distinguish between the numeral “2″, which is a formal symbol designed to be manipulated according to formal rules, and the noun “two”, which appears to name something

OK... I honestly can't comprehend how someone could simultaneously believe that "2" is just a symbol and that "two" is a blob of pure meaning. It suggests the inferential distances are actually pretty great here despite a lot of surface similarities between what Landsburg is saying and what I would say.

Instead, the point of the Peano axioms was to model what mathematicians do when they’re talking about numbers

I'm happy with this (although the next sentence suggests I may have been misinterpreting it slightly). Another way I would put it is that the Peano axioms are about making things precise and getting everyone to agree to the same rules so that they argue fairly.

Like all good models, the Peano axioms are a simplification that captures important aspects of reality without attempting to reproduce reality in detail.

I'd like an example here.

I’d probably start with something like Bertrand Russell’s account of numbers: We say that two sets of objects are “equinumerous” if they can be placed in one-one correspondence with each other

This is defining numbers in terms of sets, which he explicitly criticizes Yudkowsky for later. I'll take the charitable interpretation though, which would be "Y thinks he can somehow avoid defining numbers in terms of sets... which it turns out he can't... so if you're going to do that anyway you may as well take the more straightforward Russell approach"

the whole point of logic is that it is a mechanical system for deriving inferences from assumptions, based on the forms of sentences without any reference to their meanings

I really like this definition of logic! It doesn't seem to be completely standard though, and Yudkowksy is using it in the more general sense of valid reasoning. So this point is basically about the semantics of the word "logic".

But Yudkowsky is trying to derive meaning from the operation of inference, which won’t work because in second-order logic, meaning comes first.

I think this is a slight strawmanning of Yudkowsky. Here Yudkowsky is trying to define the meaning of one particular system - the natural numbers - not define the entire concept of meaning.

he’s effectively resorted to taking sets as primitive objects

So when studying logic, model theory etc. we have to make a distinction between the system of reasoning that we are studying and "meta-reasoning". Meta-reasoning can be done in natural language or in formal mathematics - generally I prefer when it's a bit more mathy because of the thing of precision/agreeing to play by the rules that I mentioned earlier. I don't see math and natural language as operating at different levels of abstraction though - clearly Landsburg disagrees (with the whole 2/two thing) but it's hard for me to understand why.

Anyway, if you're doing model theory then your meta-reasoning involves sets. If you use natural language instead though, you're probably still using sets at least by implication.

full-fledged Platonic acknowledgement

I think this is what using natural language as your meta-reasoning feels like from the inside. Landsburg would say that the natural numbers exist and that "two" refers to a particular one; in model theory this would be "there exists a model N with such-and-such properties, and we have a mapping from symbols "0", "1", "2" etc. to elements of N".

It seems enormouosly more plausible to me that numbes are “just out there” than that physical objects are “just out there”

So he's saying that the existence of numbers implies the existence of the physical universe, and therefore that the existence of numbers is more likely?? Or is there some quality of "just out there"ness that's distinct from existence?

mathematicians from Pythagoras through Dedekind had absolutely no problem talking about numbers in the absence of the Peano axioms.

So, they were lucky. It could have been that that-which-Pythagoras-calls-number was not that-which-Fibonacci-calls-numbers.

Are there absolutely no examples of cases where mathematicians disagreed about some theorem about some not-yet-axiomatized subject, and then it turns out the disagreement was because they were actually talking about different things?

(I know of no such examples, but I would be surprised it none exist)

Is there really that much difference between Second Order Logic, and First Order Logic quantifying over sets, or over classes?

Put it another way, Eliezer is using SOL to pinpoint the concept of a (natural) "number". But to do that thoroughly he also needs to pinpoint the concept of a "property" or a "set" of numbers with that property, so he needs some axioms of set theory. And those set theory axioms had better be in FOL, because if they are also in SOL he will further need to pinpoint properties of sets (or proper classes of sets), so he needs some axioms of class theory. Somewhere it all bottoms out in FOL, doesn't it?

From Landsburg's "Accounting for Numbers," point 9:

numbers are indeed just “out there” and that they are directly accessible to our intuitions.

[Emphasis mine.]

I'm conflicted on whether to think the world is actually physical in a special way, or simply mathematical in such a way that all mathematical structures exist and are equally real. My greatest sticking point to accepting the latter position is the emphasized part of the quote.

Okay, if I grant that numbers are "out there," we do seem to interact with them via a cognitive algorithm we call "intuition." But how the heck does that work!? Nothing I know about cognitive science suggests that intuitions are somehow specially equipped to interact with numbers, or are fundamentally different than the brain's other cognitive algorithms.

The crux of the argument for a mathematical universe - so far as I understand it - is parsimony. But in order for parsimony to be the relevant criteria, both hypotheses need have the same experimental predictions. In other words, if the mathematical universe hypothesis is correct, we would expect it to be consistent with our non-mathematical universe understanding of cognitive science. But with respect to intuitions, this doesn't seem to be the case.

Without having a satisfying account of how intuitions work in a mathematical universe, the MUH isn't consistent with my current understanding of reality. Is there such an explanation of intuitions? If not, maybe a more fruitful question to ask might be:

What kind of cognitive algorithms generate the feeling that numbers are "out there"?