It seems you are talking about being self-conscious, not about language fluency.

I didn't say it was about fluency. But I don't think it's about self-consciousness, either. Native speakers of a language pick the appropriate tense and aspect forms of verbs perfectly effortlessly - or how often do you hear a native speaker of English use a progressive in a case where it strikes you as inappropriate and you would say that they should really have used a plain tense here, for example?* - while for L2 speakers, it is generally pretty hard to grasp all the details of a language's tense/aspect system.

*I'm choosing the progressive as an example because it's easiest to describe, not because I think it's a candidate for serious unacquirability. It's known to be quite hard for native speakers of a language that has no aspect, but it's certainly possible to get to a point where you don't use the progressive wrongly essentially ever.

What possible mechanism do you have in mind?

For syntax, you would really need to be a strong Chomskian to expect any such things. For semantics, it seems to be a bit more plausible a priori: maybe as an adult, you have a hard time learning new ways of carving up the world?

Well, let's get specific. Which test do you assert native speakers will pass and ESL people will not (except for the "most exceptional")?

I don't know of a pass/fail format test, but I expect reading speed and the speed of their speech to be lower in L2 speakers than in L1 speakers of comparable intelligence. I would also expect that if you measure cognitive load somehow, language processing in an L2 requires more of your capacity than processing your L1. I would also expect that the active vocabulary of L1 speakers is generally larger than that of an L2 speaker even if all the words in the L1 speaker's active lexicon are in the L2 speaker's passive vocabulary.