Non-Africans did interbreed with Neanderthals, as someone posted a few days ago. The recent article on the subject concluded that only Africans lack the Neanderthal genes.

Most genes present in Africa didn't have a chance to spread outside of Africa, because the people carrying the genes didn't spread outside of Africa. (There's a bit of chicken-and-egg to that answer.) The Sahara is a major barrier today. I don't know if it existed 20,000 years ago.

I don't know if it's known whether the diversity of genes in Africa developed historically, or recently. It's possible that the abundance of human parasites and diseases in Africa causes faster adaptation in response to them. The longer species X has been present in an area, or the higher the density of species X there, the better other species there are adapted to species X, the harder life is for species X, the faster species X evolves.