My impression is that people tend to be exposed to grammar early on in school, in the form of a lot of arbitrary-seeming rules, which do not necessarily correspond with the colloquial spoken language

And indeed, they're not supposed to. "Grammar" in the sense of school consists of rules for signaling high status via speaking and writing. (The level of "arbitrariness" is what you'd expect given this.) Nothing to do with "grammar" in the sense of theoretical linguistics.

In that sense, language is value-neutral.

Linguists, however, are too hasty to jump to this conclusion in their attempt to explain that "evaluating" different language varieties is not their subject as linguists. There may be legitimate arguments (aesthetic, utilitarian, etc.) for why some forms of language are "better" than others; it's just that such arguments are strictly irrelevant from the point of view of theoretical linguistics (though not sociolingustics, etc).