It is absolutely the fact that gene drift is more common than mutation. Indeed, a major reason for sexual reproduction is to provide alternate genes that can mask other genes broken by mutations.

Both have nothing to do with natural selection. Genetic drift is when a gene get's lucky and spreads to the whole population even though it provides no advantage. Alternatively a gene like human vitamin C enzymes that's useful but for which there isn't strong selection pressure can die in gamblers ruin.