There's a selection effect, too. If Britain holds places that are geographically valuable (harbors in trade lanes, natural resources, etc), and geographically valuable places tend to do better, then Britain can wind up having their former colonies do better without causing it. In other words, what caused the former colony to do well made Britain colonize them, and not vice versa.

By analogy, Harvard graduates better students without providing a commensurate level of education by the simple application of strict entry requirements.