On LW, 'singularity' does not refer to a mathematical singularity, and does not involve or require physical infinities of any kind. See Yudkowsky's post on the three major meanings of the term singularity. This may resolve your physicist friend's disagreement. In any case, it is good to be clear about what exactly is meant.

Lack of cheap energy.

Ecological disruption.

Diminishing returns of computation.

Diminishing returns of engineering.

Inability to precisely manipulate matter below certain size thresholds.

All sorts of 'boring' engineering issues by which things that get more and more complicated get harder and harder faster than their benefits increase.

So my question is: What are the 'obvious' candidates for limits that take over before the all optimizable is optimized by runaway technology?

There aren't any that I'm aware of, except for "a disaster happens and everyone dies," but that's bad luck, not a hard limit. I would respond with something along the lines of "exponential growth can't continue forever, but where it levels out has huge implications for what life will look like, and it seems likely it will level out far above our current level, rather than just above our current level."

One calculation per planck time per cubic planck length in the future light cone.