Imagine that there is some behavior called "snarf," which 10% of the population thinks is morally acceptable but 90% thinks is abhorrent. The desire of the 10% to perform snarf is going to be outweighed by the desire of the 90% to ban snarf. Thus, society will not move in the direction of legalizing snarf. If that's still too abstract, substitute "gay marriage" for "snarf."

Similarly, meat eaters would probably not agree that vegetarians hold the high ground.