He touches on that in "Consider Phlebas". Basically there were other technologically advanced civs with different values, not everyone wanted to join the Culture.

Another possible explanation is that the ship Minds controlled immigration in non-obvious ways (memetic weaponry).