just part of the set of all ideas

For a start, we'd want to abandon the idea that mathematical structures are merely "ideas". A mathematician can have an idea of a structure, but the same abstract structure can often be conceived of in many different ways, and some structures are too complicated to be conceived of at all (e.g. a non-principal ultrafilter).

implied by some rules of mathematics

A structure (like the set of natural numbers together with its arithmetical operations) is not the same thing as a proposition (like "2+2=4" or "addition of natural numbers is commutative"). Structures satisfy propositions, and it may or may not be possible to systematically investigate the propositions satisfied by a structure by setting out 'axioms' and 'rules of inference' (both of which I suppose you'd call "rules of mathematics").

but not existing in any way that 2+2=4 does not exist

Better to say "not existing in any way that the numbers themselves don't exist".

how would we, as part of the answer to a math problem, know the difference between that and 'really existing'?

The real question here is "how is it possible for a mathematical structure to contain an intelligent observer?" Once you have an intelligent observer they can in principle teach themselves logic and mathematics, which will entail finding out about mathematical structures other than the one they're inhabiting.