How do socially-constructed concepts work?!
Negative example: trees. Trees exist, and trees are not socially constructed. An alien AI observing Earth from behind a Cartesian veil would be able to compress its observations by formulating a concept that pretty closely matches what we would call tree, because the atom-configurations we call "trees" robustly have a lot of things in common: once the AI has identified something as a "tree" by observing its trunk and leaves, the AI can make a lot of correct predictions about the "tree" having roots, this-and-such cellular walls, &c. without observing them directly, but rather by inference from knowledge about "trees" in general.
Positive example: Christmas. Christmas exists. An alien AI observing Earth from behind a Cartesian veil would be able to make better predictions about human behavior in many places around Epoch time 62467200 ± 31557600·n (for integer n ) by formulating the concept of "Christmas". However, Christmas is socially constructed: if humans didn't haven't a concept of "Christmas", there would be no Christmas (the AI-trick for improving predictions using the idea of "Christmas" would stop working), but if humans didn't have a concept of trees, there would still be trees (the AI-trick for improving predictions using the idea of "trees" would still work).
Semi-positive example: adulthood. Adulthood exists. There's a Sorites situation on exactly how old a human has to be to be an "adult", and different human cultures make different choices about where to draw that line. But this isn't just a boring Sorites non-problem, where different agents might use different communication signals without disagreeing about about the underlying reality (like when I say it's "hot" and you say it's "not hot, just warm" and our friend Hannelore says it's "heiß", but we all agree that it's exactly 303.6 K): an alien AI observing Earth from behind a Cartesian veil can make better predictions about whether I'll be allowed to sign contracts by reasoning about whether my Society considers me an "adult", not by directy using the simple measurement test that Society usually uses to make that determination, with exceptions like minor emancipation.
My work-in-progress take: an agent outside Society observing from behind a Cartesian veil, who only needs to predict, but never to intervene, can treat socially-constructed concepts the same as any other: "Christmas" is just a pattern of behavior in some humans, just like "trees" are a pattern of organic matter. What makes social construction special is that it's a case where a "map" is exerting control over the "territory": whether I'm considered an "adult" isn't just putting a semi-arbitrary line on the spectrum of how humans differ by age (although it's also that); which Schelling point the line settles on is used as an input into decisions—therefore, predictions that depend on those decisions also need to consider the line, a self-fulfilling prophecy. Alarmingly, this can give agents an incentive to fight over shared maps!
One we already talked about together is the problem of defining the locality of goals. From an abstraction point of view, local goals (goals about inputs) and non-local goals (goals about properties of the world) are both abstractions: they throw away information. But with completely different results!
This plays well with impact measures, too. I can definitely include it.