I usually interpret the intended difference to have to do with the relevance of belief in observers.

That is, when my X is described as "merely an act," I understand the speaker to be suggesting that if I believed myself unobserved, I would not demonstrate X. (There's also a related implication having to do with sincerity, but that's much trickier to express in a succinct coherent way.)

Of course, as you point out, in reality it's more complicated than that, and something can be in some meaningful sense "an act" while also being something I do for my own benefit.