Yes and no - the whole discussion is actually two discussions, I think.

One is about in-principle possibility, the presence of something like a critical period, etc. There it is crucial for confounders.

The second discussion is about in-practice possibility, whether people starting later can reasonably expect to get to the same level of proficiency. Here the "confounders" are actually part of what this is about.