I don't think this interpretation can hold up: the body of titotal's post doesn't deal with the good vs bad timeline. It's just about the uncertainty of modelling AI progress which applies for both the good and bad timelines.

I think it's an intentional pun, like, "whether forecasters" are people who predict whether something will happen or not.

What about tuning the fiddle strings down 1 tone?

You say this:

If you’re thinking, “Wait no, I’m pretty sure my group is fundamentally about X, which is fundamentally good,” then you’re probably still in Red or Blue.

But you also say this:

First, the Grey tribe is about something, [...] things that people already think are good in themselves.

Doesn't the first statement completely undermine the second one?

I guess you meant jukebox, not jutebox. Unless there is some kind of record-playing box made of jute fiber that I haven't heard of...

but I recently tried again to see if it could learn at runtime not to lose in the same way multiple times. It couldn't. I was able to play the same strategy over and over again in the same chat history and win every time.

I wonder if having the losses in the chat history would instead be training/reinforcing it to lose every time.

Yes, my understanding is that the system prompt isn't really priviledged in any way by the LLM itself, just in the scaffolding around it.

But regardless, this sounds to me less like maintaining or forming a sense of purpose, and more like retrieving information from the context window.

That is, if the LLM has previously seen (through system prompt or first instruction or whatever) "your purpose is to assist the user", and later sees "what is your purpose?" an answer saying "my purpose is to assist the user" doesn't seem like evidence of purposefulness. Same if you run the exercise with "flurbles are purple", and later "what color are flurbles?" with the answer "purple".

#2: Purposefulness.  The Big 3 LLMs typically maintain or can at least form a sense of purpose or intention throughout a conversation with you, such as to assist you.

Isn't this just because the system prompt is always saying something along the lines of "your purpose is to assist the user"?