I think it has to do with the "leaving the tribe" aspect more than anything. Those of us who became devout in one of the more serious religions (that is, religions that view everyone else as a spectrum from "good but deeply flawed" to "hellbound") had that religion encompass most of our social world, and so in order to leave it we had to face the prospect of ostracism from all the people we cared about. The evolutionary pressures to never get ostracized make for a lot of subconscious bias to fight, and a pretty dramatic tale.

If your conversion was undramatic, therefore, I conjecture that you didn't have lots of friends or family who might have abandoned you if you stopped being religious.