Presumably they disappear somehow (old age?) as well,

Do I really need to quote the bit where Quirrell talks about how they are "undying"? I rather highly doubt that the shadows of Death die of old age.

since Harry isn't asking them questions from the days of Atlantis. We know that they existed at least as far back as the founders' days, and he's also not pumping them for info about magic from

Harry can't hear them speak. He doesn't even think they have anything like minds or memories, either.

Plus I can't imagine hundreds of idiots summoning dementors and nobody figuring out that they're the manifestation of death.

Do we have any indication that more than one person has performed the ritual to summon them? Maybe all the Dementors that exist today were created by one madman in Atlantis.

Edit: Or, here's a thought: suppose some incautious Dark Wizard performs the ritual to summon Death. The end result of which is - oops! - he's suddenly face-to-nothingness with a Dementor. In all likelihood the thing's first instinct is to give him a big ol' smooch in gratitude.