Asking the question "to whom am I thankful", more than feeling thankfulness, is the superstitious behavior here. If we allow thankfulness to have no object, the conflict is resolved.
You may be thankful to the other driver. You may be thankful to yourself or the relevant parts of your body that helped avoid a worse outcome. You may be thankful to your past self, or your alternative universe self. You may be thanking the laws of physics. Maybe we can just drop the to preposition and be done with that.
As to why you are feeling thankful, I find Antisuji and DaFranker explanations plausible.
As to what should be done about it, I can't identify negative consequences to being thankful. Do you suspect it trains you to for risk or clumsiness?
Well, I used to think that I do not believe in anything supernatural that affects what happens to me, but I'm wondering if maybe I actually do alieve in it. For example, a few days ago I had a close call in traffic, and when a collision I fully expected to happen just a second prior did not transpire, I mentally thanked... whom? I definitely had a clear feeling of gratitude for escaping, and I don't normally mean it literally when I say "Thank God!". So, who or what did I feel thankful to? I've never been religious, and I got rid of most of my superstitions over the years, but apparently there is still something there, and I do not know how to react to this knowledge.
What would be the proper reaction after a close call? Shrug and say "got lucky this time, should be more cautious next time"? What about when waiting for a diagnosis, what does sort-of-praying "please, please, let everything be OK" say about one's true beliefs? I know that I am much better at not blaming the world when something bad happens to me by chance than at not thanking the world when something good happens. Should it not be symmetric? Which part of a normally non-religious person wakes up and asserts itself in a crisis situation out of their control? Should it be embraced, suppressed, worked on?