There are a lot of issues with ontologically committing yourself to possible worlds (just look up modal realism and any W.V.Quine essay on it) that are beyond the scope of this post. I'm not saying that we shouldn't use phrases like "there are propositions that are just false, the worlds they are true about don't exist" but those propositions do imply that possible worlds exist. In my formulation, I take special care not to ontologically commit myself to possible worlds. To avoid that commitment, I always treat possible worlds as if they are parts of a model, not of the modeled:

Both Johnny and Sally are undecided about "The top card is a heart."; their model of the world splits at that point of representation.

Each agent has its own unique set of possible worlds it thinks it might find itself in, and it's determined by that agent's background knowledge. Maybe I should use the phrase "Hypothetical world", or "imaginable world" instead of "Possible world" to make that clearer.

I added this sentence thinking about your comment, thanks for the help:

To be clear, we don't need to believe that possible worlds actually exist to accept this view of belief; we just need to believe that any agent capable of being undecided about a proposition is also capable of imagining alternative ways the world could consistently turn out to be, i.e., capable of imagining possible worlds.