This doesn't solve the problem of motivation to lie about (or change) one's utility function to move the group equilibrium, does it?

My rules are:

1. Unlabeled walls of AI-generated text intended for humans are never okay.
2. If the text is purely formalized or logistical and not a wall, that can be unlabeled.
3. If the text is not intended to be something a human reads, game on.
4. If the text is clearly labeled as AI that is fine if and only if the point is to show that the information comes from a neutral third party of sorts.

I agree, but these are too specific.  This is a violation of discourse norms, and doesn't need to be separated into ai-generated vs human-nerdsplaining walls of text.   Also, it's always been annoying: LMGTFY.

I think most relationships and values are multi-dimensional and don't collapse very easily into this dimension.  

Separately, I generally dislike and cannot use models that diverge so far from reality - this situation does not come up, and if it did there would be no certainties that you posit.  

Crux identification is already an unsolved problem in many debates (including internal truth-seeking).  I strongly expect that categorization of "primary" and "secondary" does not reduce the difficulty in any way.

An example would help a lot.

Are you including humans in your thinking on this?  I'm generally suspicious of heavy analogizing between human and LLM, but for affect and qualia ("what if feels like") it's probably the best we have.

Learning, especially cultural learning, and especially in children, and super-especially in neurodivergent people, can be unpleasant in various ways.  We generally think it's worth imposing this pain on people, in order to increase their conformity and ability/willingness to cooperate with us, and generally adults say it was the right choice.

How much of this analogizes to LLMs?  I love that you're trying to measure the stress and pain of training, but I hope you're putting some effort in to defining the value that results (that the LLM actually gets executed, rather than being thrown away) as well.

Good to think about this and try to put together scenarios to identify interesting potential levers or just signs that we might react to. 

Unfortunately, this relies on some unstated assumptions about what "cooperation" actually means, and how resource control works - what is the pivot from companies controlling resources (and growing based on investment/public-support) to the AIs controlling them directly and growing based on ... something else?

At these scales, "ownership" becomes a funny concept.  Current mechanisms for acquiring more compute and for sabotoging or assisting your competitors are unlikely to hold for more than a few more doublings.

Almost all of the obvious interventions (excercise, at least basic medical and dental care, non-pathological eating, and reasonable limits on nootropic and recreational drugs or alcohol) have massive short- and medium-term benefits.  One doesn't need to do much long-term math or have anywhere near calibrated estimates to understand the EV of these things.

Fully optimizing toward longevity and away from enjoyment and ease in the short-term may be correct, for some estimates, but it's not as clear.  And irrelevant for the vast majority of daily behaviors.

My only surprise is that this is surprising in any way.  I'd put the numbers at:

• 20-30 percent will actively support most things the current leadership does
• 30-40 percent will go along with it, even if they don't particularly like it
• 10-20 percent will complain about the worst disagreements, but not take significant action that puts their lifestyle at risk.
• 2-10 percent will do anything beyond attending a protest (like moving their person or assets to a different jurisdiction, or changing jobs or running for opposing office)
• 5-20 percent lizardman

Fortunately, most democratic systems have time-based checkpoints where the threshold for change is a lot lower (people in the 2nd and 3rd groups can make change more easily during elections), and the activist numbers tend to be MUCH higher for policies that hugely impact that cadence.   

thanks for the conversation, I'm bowing out here.   I'll read further comments, but (probably) not respond.  I suspect we have a crux somewhere around identification of actors, and mechanisms of bridging causal responsibility for acausal (imagined) events, but I think there's an inferential gap where you and I have divergent enough priors and models that we won't be able to agree on them.