Let A(X) = “There are plenty of non X-ers who think it's immoral for anyone to X, whereas there aren't many X-ers who think it's immoral for other people to refuse to X.”

Let B(X) = “People who are non-X-ers usually are because of a cached belief, whereas people who are X-ers usually are because they've thought about both possibilities and concluded one is better.”

Are you really saying that log(P(A(X)|B(X))/P(A(X)|¬B(X))) ≤ 0? or do you just mean that while positive it is very small? Because I really can't see how A(X) can be more likely given ¬B(X) than given B(X).