For sure you can't have cross-entropy constrained by both of $H_P$ and $H_Q$ if they [ETA that is, $P$ and $Q$, not their entropies] differ at all. Maybe you'd make some headway if you introduced some slackness into the entropy constraints? Or maybe this is a deadend. Or maybe I'm not following right.

I think this reveals that the maxent problem you post at the start (maximum entropy constrained by cross-entropy not exceeding entropy) is actually fully specified by the constraint, rather than being sensitive to the expression being maximised. So the maxent is irrelevant. Don't know what downstream implications that has.

Just a note, your notation $E_P[-\log Q[X\left]\right]$ is the cross-entropy $H(P,Q)$ which indeed (Gibbs) is minimised (for given $P$) when $Q=P$.

I sometimes (since 2022^[1]) say that I'd prefer if the widespread LM products 'spoke in wikipedia voice' rather than being specifically trained to put forward a facade of personhood (first person pronouns etc), which is especially geared to bypassing rational engagement. I don't think people mostly thought of LMs as 'the AIs' until chat posttraining hit it big.

Sadly I think that's by default a losing battle as the chat interface is both especially navigable (as you mention) and intuitively appealing to a mass market.

Note that we're aware of the cautionary tales of Cyc, Xanadu, and Arbital!

We're hoping a combination of:

• more minimalistic
• more open and interfaceable
• semantically sensitive software components (LMs)
• cheaply available clerical labour (LMs)

means that 'this time is different'.

At FLF one of our initiatives we're recruiting for is an 'epistemic stack', which I think fits the bill as a backend/foundation for many of the desiderata you're describing. LLM chat interface would be one UX form factor on top.

Epistemic stack would be a (probably distributed) cache of annotations and metadata connecting claims to supporting sources, constructable and expandable dynamically. The cheapest and widest-coverage construction would use LM-based agents over webtext, inferring support from existing links, citations, and sometimes proactive web search for supporting (or contradictory) sources. Human participants (authors and readers) would be able to provide various annotations, including endorsement of (or alterations to) inferred epistemic support links. Something like git, versioned, signed, annotated DAGs would then be available to downstream epistemic applications (including via RAG for LM consumption, but also many other imaginable formats).

MVP on-demand tree construction for a given claim is already practical, though unreliable and more expensive than a system with caching would be.

Down the line, if more verifiable sources of ground data (signed cameras, etc.) get more widespread, such data would readily integrate as leaves.

Compare also Society Library which has some similar prior work (mostly manual), and may be moving in a similar direction.

There has also been related discussion of 'wikipedia for LLMs', and though I haven't heard more technical refinement from such proponents, that term might be intended to expand to a similar overall concept.

Note that 'wikipedia by LLM', like grokipedia, does not currently have any meaningful claim to epistemic grounding, correctability, or transparency/legibility, though its form factor would at least inherit the navigability of wikipedia.

a 2×2 table for when

I think the friend/enemy axis probably works more like a scalar coefficient (or has many more than two categories). But otherwise I'm really interested to see you (who've put much more effort in here) converge on a similar model to my much rougher one. Does that add up for you, or do you think something else is happening? (I could also imagine a model with two coefficients, one positive, one negative, which could distinguish 'frenemy' from 'stranger'.)

Thanks for this post! (It's a reminder I keep wanting to read more of your stuff and not making time for it.)

See also Deferring from Cotton-Barratt (2022)

You might defer because you think they know better (epistemic deferring), or because there is a formal or social expectation that you should go along with their view (deferring to authority).

Both types of deferring are important — epistemic deferring lets people borrow the fruits of knowledge; deferring to authority enables strong coordination. But they are two-edged.

...when someone defers without everyone understanding that's what's happening, it can cause issues. Similarly, unacknowledged expectations of deferral from others can cause problems.

My rough recollection is that in several early mesopotamian civilisations, much land (a majority?) and property was 'owned' by gods, for whom the various classes worked (priests administrating, peasants farming, and so on).

ETA Britannica has a discussion:

They constituted, as it were, a landed nobility, each god owning and working an estate—his temple and its lands

A bugbear of mine is being told to be surprised or astonished. Especially when I wouldn't be, but even when I am. It's not up to you, author! This post is riddled with that particular dark art, which I perceived to be on the rise in LW.