Somewhere in the books, I think in Look to Windward, it's pointed out that ambassadors to the Culture end up assimilated and serving unofficially as ambassadors for the Culture instead.

Also, the Culture appears to have trillions of citizens. You could have a large amount of immigration and still not see much of the impact.