I think philosophers who think that the categoricity of second-order Peano arithmetic allows us to refer to the natural numbers uniquely tend to also reject the causal theory of reference, precisely because the causal theory of reference is usually put as requiring all reference to be causally guided. Among those, lots of people more-or-less think that references can be fixed by some kinds of description, and I think logical descriptions of this kind would be pretty uncontroversial.

OTOH, for some reason everyone in philosophy of maths is allergic to second-order logic (blame Quine), so the categoricity argument doesn't always hold water. For some discussion, there's a section in the SEP entry on Philosophy of Mathematics.

(To give one of the reasons why people don't like SOL: to interpret it fully you seem to need set theory. Properties basically behave like sets, and so you can make SOL statements that are valid iff the Continuum Hypothesis is true, for example. It seems wrong that logic should depend on set theory in this way.)

There is a lot of psuedo-arguing going on here. I'm a mathematician who specializes in logic and foundations, I think I can settle most of the controversy with the following summary:

1) Second-order logic is a perfectly acceptable way to formalize mathematics-in-general, stronger than Peano Arithmetic but somewhat weaker than ZFC (comparable in proof-theoretic strength to "Type Theory" of "Maclane Set Theory" or "Bounded Zermelo Set Theory", which means sufficient for almost all mainstream mathematical research, though inadequate for results which depend on Frankel's Replacement Axiom).

2) First-order logic has better syntactic properties than Second-order logic, in particular it satisfies Godel's Completeness Theorem which guarantees that the syntax is adequate for the semantics. Second-order logic has better semantic properties than first-order logic, in particular avoiding nonstandard models of the integers and of set-theoretical universes V_k for various ordinals k.

3) The critics of second-order logic say that they don't understand the concept of "finite integer" and "all subsets" and that the people using second-order logic are making meaningless noises, and all we really can be sure about is ZFC and its consequences, and since the ZFC axioms actually allow us to prove more theorems about objects we care about than the usual formalizations of second-order logic (such as, for example, the consistency of full Zermelo set theory), the fans of second-order logic should just quit and use ZFC instead

4) the fans of second-order logic reply that nothing we actually prove and translate into the language of set theory is false and our reasoning is sound by your standards so stop insulting us by claiming that since YOU can't decide what's true about sets, OUR notion of "property" or "subset" is vague and incoherent. We say that CH is either true of false but we don't know which because we don't have a deductive calculus that settles it which we are confident in, you say that CH is vague and meaningless, but this is a pseudo-problem until either you come up with new set-theoretical axioms which decide CH or we come up with a stonger deductive system which decides it. We can still prove "ZF proves Con(Z)", we just think that the theorems we can derive in pure second-order logic are epistemologically better supported than theorems you need the full power of ZFC to prove (because they already follow from "logical" rather than "mathematical" axioms) and you shouldn't mind us making this distinction.

5) The best theories of physics we have need real infinities and finitely many iterations of the Powerset axiom but Second Order Logic or "Type Theory" is completely adequate for them and it is hard to conceive that questions about higher infinites that require infinitely many iterations of the Powerset operation can be relevant to physics.

6) It is also hard to conceive that any form of mathematical finitism is adequate for making actual progress in theoretical physics, even if results that are proved using standard infinitary mathematics can be reformulated as finitary statements about integers and computations, these nominalistic reductions are so cumbersome that the self-imposed avoidance of infinitary ontology cripples physical insight.

This prompted a memory of something I read in one of my undergrad psychology books a few years ago, which is probably referencing the same study, though using two different examples and one the same as the above example (though the phrasing is slightly different). Here is the extract:

Hindsight (After-the-Fact understanding)

Many people erroneously believe that psychology is nothing more than common sense. "I knew that all along!" or "They had to do a study to find that out?" are common responses to some psychological research. For example, decades ago a New York Times book reviewer criticized a report titled The American Soldier (Stouffer et al., 1949a,1949b), which summarized the results of a study of the attitudes and behavior of U.S. soldiers during World War II. The reviewer blasted the government for spending a lot of money to "tell us nothing we don't already know."

1. Compared to White soldiers, Black soldiers were less motivated to become officers.

2. During basic training, soldiers from rural areas had higher morale and adapted better than soldiers from large cities.

3. Soldiers in Europe were more motivated to return home while the fighting was going on than they were after the war ended.

You should have no difficulty explaining these results. Typical reasoning might go something like this: (1) Due to widespread prejudice, Black soldiers knew that they had little chance of becoming officers. Why should they torment themselves wanting something that was unattainable? (2) It's obvious that the rigors of basic training would seem easier to people from farm settings, who were used to hard work and rising at the crack of dawn. (3) Any sane person would have wanted to go home while bullets were flying and people were dying.

Did your explanations resemble these? If so, they are perfectly reasonable. There is one catch, however. The results of the actual study were the opposite of the preceding statements. in fact, Black soldiers were more motivated than White soldiers to become officers, city boys had a higher morale than farm boys during basic training, and soldiers were more eager to return home after the war ended than during the fighting. When told these actual results, our students quickly found explanations for them. In short, it is easy to arrive at reasonable after-the-fact explanations for almost any result.

Source:Pass, M. W. & Smith, R.E. (2007) Psychology:The Science of Mind and Behavior (Third Edition). McGraw HIll: Boston, pages 31-32

In hindsight, I guess I must have known that it would be a good idea to hang on to my undergrad textbooks. Or did I?