In your comment that you link to, you give a more narrow definition, specifying "the scientific method". I agree there might be things outside of that (which will undoubtedly be absorbed into accepted science over time, mutating the concepts of the scientific method to suit new knowledge).

But here you specify all "statements about the world". In that case I can say outright that in no meaningful sense does there "exist" something not in the world which cannot interact with the world. By the generalized p-zombie principle: if it cannot interact with us, then it is not causally involved with your reason for speaking about it. Nothing you will ever think or do or say or believe in, or perceive with your senses, will be causally related to something outside "the world". So there is no reason to ever discuss such a thing.

Further, math (logic) is in the world. It does not have some Platonic independent "existence" because existence is a predicate of things in the physical world; it makes as much sense for a pure circle to exist as to not exist.

The reason we talk about math is that it is lawfully embodied in the physical world. Our brains are so built as to be able to think about math. When we think about math we find that we enjoy it, and also that we can use it for useful purposes of applied science. So we keep talking more about math. That is a complete explanation of where math comes from. No additional postulate of math "objectively existing" is required or indeed meaningful.