Part of the problem turns out to be equivocation on "wrong". Consider the position of a chocolate-lover living in a hypothetical society where 1% of people are chocolate-lovers, 89% of people are not chocolate-lovers but are tolerant of people who are, and 10% of people think that anyone who eats chocolate should be shunned, persecuted, fired, exiled, tortured, or what-have-you.

Eating chocolate is not wrong according to our standards, nor by their own, nor by those of the majority of their society. However, many chocolate-eaters in this hypothetical society would prefer to cover up their actions, not out of shame but out of risk avoidance. They don't want to be mistreated by the anti-chocolate people who outnumber them ten to one. In the 1 in 10 chance that their boss is anti-chocolate, they don't want to be fired or mistreated on the job. And so on.

In effect, the saying should be, "If you haven't done anything that anyone anywhere disapproves of, you have nothing to hide."