Given the way the term is actuallly used, I wouldn't call that "signalling" because "signalling" normally refers to demonstrating that you have some trait by doing something other than performing the trait itself

This seem to be the meaning common on these boards, yes.

And taking a math test to let people know that you're good at math is not signalling, but going to a mathematicians' club to let people know that you're good at math may be signalling.

Going to mathematicians club (and the like) is something that you can do if you aren't any good at math, though. And it only works as a "signal" of being good at math because most people go to that club for other reasons (that would be dependent on being good at math).

Signalling was supposed to be about credibly conveying information to another party whenever there is a motivation for you to lie.

It seems that instead signalling is used to refer to behaviours portrayed in "Flowers for Charlie" episode of "It's always sunny in Philadelphia".