"There's a lot of confusion here."

Yes, and I'd say the biggest confusion is construing Mary's promiscuity as an accidental, gratuitous part of the joke. The whole point of the joke is that such promiscuity would be the first, most natural inference to draw were a self-proclaimed virgin to tell us that she'd been made pregnant by God.