I don't necessarily disagree with what you literally wrote. But also, at a more pre-theoretic level, IMO the sequence of events here should be really disturbing (if you haven't already been disturbed by other similar sequences of events). And I don't know what to do with that disturbedness, but "just feel disturbed and don't do anything" also doesn't seem right. (Not that you said that.)

(I think I'll go with "alloprosphanize" for now... not catchy but ok. https://tsvibt.github.io/theory/pages/bl_25_10_08_12_27_48_493434.html )

Exoclarification? Alloclarification? Democlarification (dēmos - "people")?

These are good ideas, thanks!

Ostentiation:

So there's steelmanning, where you construct a view that isn't your interlocutor's but is, according to you, more true / coherent / believable than your interlocutor's. Then there's the Ideological Turing Test, where you restate your interlocutor's view in such a way that ze fully endorses your restatement.

Another dimension is how clear things are to the audience. A further criterion for restating your interlocutor's view is the extent to which your restatement makes it feasible / easy for your audience to (accurately) judge that view. You could pass an ITT without especially hitting this bar. Your interlocutor's view may have an upstream crux that ze doesn't especially feel has to be brought out (so, not necessary for the ITT), but which is the most cruxy element for most of the audience. You can pass the ITT while emphasizing that crux or while not emphasizing that crux; from your interlocutor's point of view, the crux is not necessarily that central, but is agreeable if stated.

A proposed term for this bar of exposition / restatement: ostentiate / ostentiation. Other terms:

• ekthesize / exthesize; phosphanize, epiphotize; elogize; anaply / anaplain; anaptychize / exptychize / eptychize;
• adluminate, superluminate; ostentiate, superostentiate;
• belight, beshine, enbeacon.

Cf. https://web.archive.org/web/20120331071849/http://www.paul-almond.com/WhatIsALowLevelLanguage.htm

is a fine example of thinking you get when smart people do evil things and their minds come up with smart justifications why they are the heroes

I want to just agree because fuck those guys, but actually I think it's also shit justifications. A good justification might come from exploring the various cases where we have decided to not make something, analyzing them, and then finding that there's some set of reasons that those cases don't transfer as possibilities for AI stuff.

This seems fine and good--for laying some foundations, which you can use for your own further theorizing, which will make you ready to learn from more reliable + rich expert sources over time. Then you can report that stuff. If instead you're directly reporting your immediately-post-LLM models, I currently don't think I want to read that stuff, or would want a warning. (I'm not necessarily pushing for some big policy, that seems hard. I would push for personal standards though.)

(I don't know to what extent I'm agreeing with Duncan's statements here. I separately wrote comments about something like this new CFAR, though I'm not sure how much overlap there is between this vs. what I was commenting about. One thing my comments brought up was that I felt earlier CFAR things were, besides being often very helpful to many people including me, also not infrequently damaging to epistemics for group-related reasons, which seems possibly related to the "status differential" thing Duncan alludes to downthread. My comments touched on how CFAR in general and maybe Anna in particular did not nearly sufficiently treat incoming people as authors of the group narrative, and that this creates lots of distortionary effects (see for example https://www.lesswrong.com/posts/ksBcnbfepc4HopHau/dangers-of-deference ). And, similar to what Duncan says, this felt to me like a hydra / whack-a-mole type problem, where multiple patches didn't address the underlying thing. Though, with that type of problem, it tends to be hard to accurately characterize (and judge and fix-or-endorse) the underlying thing.)

There's no such thing as "a domain where LLMs are particularly likely to hallucinate". In every domain there's some obscure jagged boundary, not very far from normal standard questions to ask, where LLMs will hallucinate, usually plausibly to a non-expert.

I am treating LLM output as somewhat less trustworthy than I would trust what a colleague of mine says, but not fundamentally different.

If you're asking a human about some even mildly specialized topic, like history of Spain in the 17th century or different crop rotation methods or ordinary differential equations, and there's no special reason that they really want to appear like they know what they're talking about, they'll generally just say "IDK". LLMs are much less like that IME. I think this is actually a big difference in practice, at least in the domains I've tried (reproductive biology). LLMs routinely give misleading / false / out-of-date / vague-but-deceptively-satiating summaries.