If I remember my Lakoff & Núñez correctly, they were arguing that even the most abstract and un-physical-seeming of maths is constructed on foundations that derive from the way we perceive the physical world.

Let me pick up the book again... ah, right. They define two kinds of conceptual metaphor:

1. Grounding metaphors yield basic, directly grounded ideas. Examples: addition as adding objects to a collection, subtraction as taking objects away from a collection, sets as containers, members of a set as objects in a container. These usually require little instruction.
2. Linking metaphors yield sophisticated ideas, sometimes called abstract ideas. Examples: numbers as points on a line, geometrical figures as algebraic equations, operations on classes as algebraic operations. These require a significant amount of explicit instruction.

Their argument is that for any kind of abstract mathematics, if you trace back its origin for long enough, you finally end up at some grounding and linking metaphors that have originally been derived from our understanding of physical reality.

As an example of the technique, they discuss the laws of arithmetic as having been derived from four grounding metaphors: Object Collection (if you put one and one physical objects together, you have a collection of two objects), Object Construction (physical objects are made up of smaller physical objects; used for understanding expressions like "five is made up of two plus three" or "you can factor 28 into 7 times 4"), Measuring Stick (physical distances correspond to numbers; gave birth to irrational numbers, when the Pythagorean theorem was used to prove their existence by assuming that there's a number that corresponds to the length of the hypotenuse), and Motion Along A Path (used in the sixteenth century to invent the concept of the number line, and the notion of a number as lying between two other numbers).

Now, they argue that these grounding metaphors, each by themselves, are not sufficient to define the laws of arithmetic for negative numbers. Rather you need to combine them into a new metaphor that uses parts of each, and then define your new laws in terms of that newly-constructed metaphor.

Defining negative numbers is straightforward using these metaphors: if you have the concept of a number line, you can define negative numbers as "point-locations on the path on the side opposite the origin from positive numbers", so e.g. -5 is the point five steps to the left of the origin point, symmetrical to +5 which is five steps to right of the origin point.

Next we can use Motion Along A Path to define addition and subtraction: adding positive numbers is moving towards the right, addition of negative numbers is moving towards the left, subtraction of positive numbers is moving towards the left, and subtraction of negative numbers is moving towards the right. Multiplication by a positive number is also straightforward: if you are multiplying something by n times, you just perform the movement action n times.

But multiplication by a negative number has no meaning in the source domain of motion. You can't "do something a negative number of times". A new metaphor must be found, constrained by the fact that it needs to fit the fact that we've found 5 * (-2) = -10 and that, by the law of commutation (also straightforwardly derivable from the grounding metaphors), (-2) * 5 = -10.

Now:

The symmetry between positive and negative numbers motivates a straightforward metaphor for multiplication by –n: First, do multiplication by the positive number n and then move (or “rotate” via a mental rotation) to the symmetrical point—the point on the other side of the line at the same distance from the origin. This meets all the requirements imposed by all the laws. Thus, (–2) · 5 = –10, because 2 · 5 = 10 and the symmetrical point of 10 is –10. Similarly, (–2) · (–5) = 10, because 2 · (–5) = –10 and the symmetrical point of –10 is 10. Moreover, (–1) · (–1) = 1, because 1 · (–1) = –1 and the symmetrical point of –1 is 1.

The process we have just described is, from a cognitive perspective, another metaphorical blend. Given the metaphor for multiplication by positive numbers, and given the metaphors for negative numbers and for addition, we form a blend in which we have both positive and negative numbers, addition for both, and multiplication for only positive numbers. To this conceptual blend we add the new metaphor for multiplication by negative numbers, which is formulated in terms of the blend! That is, to state the new metaphor, we must use

• negative numbers as point-locations to the left of the origin,
• addition for positive and negative numbers in terms of movement, and
• multiplication by positive numbers in terms of repeated addition a positive number of times, which results in a point-location.

Only then can we formulate the new metaphor for multiplication by negative numbers using the concept of moving (or rotating) to the symmetrical point-location.

So in other words, we have taken some grounding metaphors and built a new metaphor that blends elements of them, and after having constructed that new metaphor, we use the terms of that combined metaphor to define a new metaphor on top of that.

While this example was in the context of an obviously physically applicable part of maths, their argument is that all of maths is built in this way, starting from physically grounded metaphors which are then extended and linked to build increasingly abstract forms of mathematics... but all of which are still, in the end, constrained by the physical regularities they were originally based on:

The metaphors given so far are called grounding metaphors because they directly link a domain of sensory-motor experience to a mathematical domain. But as we shall see in the chapters to come, abstract mathematics goes beyond direct grounding. The most basic forms of mathematics are directly grounded. Mathematics then uses other conceptual metaphors and conceptual blends to link one branch of mathematics to another. By means of linking metaphors, branches of mathematics that have direct grounding are extended to branches that have only indirect grounding. The more indirect the grounding in experience, the more “abstract” the mathematics is. Yet ultimately the entire edifice of mathematics does appear to have a bodily grounding, and the mechanisms linking abstract mathematics to that experiential grounding are conceptual metaphor and conceptual blending.

Insofar as logic consists in information-preserving operations, the non-physically-applicable parts of math still correspond to the real world, in that they preserve the information about the real world which was put into formulating/locating the starting formal system in the first place.

This is what makes mathematics so wondrously powerful: formality = determinism, and determinism = likelihood functions of 0 or 1. So when doing mathematics, you get whole formal systems where the theorems are always at least as true as the axioms. As long as any part of the system corresponds to the real world (and many parts of it do) and the whole system remains deterministic, then the whole system compresses information about the real world.