Your argument is missing a step, namely the one where you show that those things really are very bad

Criminal organizations murdering thousands people are not something very bad?

even though this sort of analysis suggests that they do little harm.

It is a reductio ad absurdum.

But perhaps you are imagining a version of James that would endorse buying the drugs even if that does no good at all, merely on the grounds that the harm done is small.

Buying drugs (well, let's say marijuana) supposedly does good to the rational consumers who like them. Like buying child pornography or visiting child prostitutes in third world countries does good to pedophiles, and so on.
All these people could use marginal analysis to argue that whatever harm they are doing is negligible and doesn't outweigh their gains.

If you agree with them then clearly you have much different moral principles that I have.