No. You have a rule that hypothetically would produce an infinite set if applied ad infinitum.

Yep, exactly; no problem with that, that's how mathematics works. There is only a problem if someone wants to write down every element of an infinite set.

there is a difference between the concept of an infinite set and an actual infinite set

This is mathematics. The concept of a mathematical object is the object, because the "concept" version satisfies all the same rules (axioms) as any "actual" version, and these rules completely describe its structure, and (broadly) mathematics is the study of structure/patterns.

One does not need a physical basis for these rules, and so one does not need a physical basis for structures generated by such rules.