One often sees talk here of something called the "Mind Projection Fallacy". In essence, it is the error of taking something subjective and treating it as objective. A canonical example would be saying "X is mysterious", rather than "I don't understand X". That is, what ought to be handled as a two-place predicate IsMysteriousTo(X,N) (topic X is mysterious to person N) is erroneously handled as a one-place predicate IsMysterious(X) (topic X is mysterious, full stop).

One criticism that might be made of the position you are taking here is that you are falling into a form of the Mind Projection Fallacy. Try imagining Exists(X) as a fallacious one-place predicate and replace it with ExistsFor(X,N), i.e. X has existence from the viewpoint of observer N.

Similarly, you should say "X was instantiated at a point in time in my past, hence X currently exists from my viewpoint, and I expect X to cease to exist at some point in my future. Any other supposed X, existing on some other branch of reality, is not the X to which I refer."

This, as I understand it, is the way Kripke says that issues of identity and existence must be handled in modal logics.

If you can accept this viewpoint, then it is just a small step to realizing that the people you are disagreeing with, those who consider past, future, and all branches of reality as equally "real" are actually talking about the model theory for your modal logic.