Are there other cases where someone might do a deliberately bad job at convincing people of a falsehood, in order to filter for the most gullible or susceptible marks?

Cults!

It is also not unknown for someone with something genuine to teach to actively filter out those not capable of learning it.