All of metasemi's Comments + Replies

CS Lewis FTW.

I don't know what Lewis thought about the bomb, but I trust he would have been all for trying to avert nuclear calamity. Such a belief would have taken nothing away from the wisdom of the passage you quoted. We should reason as hard as we can about the future and strive for the best outcomes, but the universe wants to unfold, will continue to unfold, and will never oppress us with certain knowledge of our greater fate: uncertainty is the human condition. Therefore we should bestow on the generations that follow us optimism, resilience, agency, and when we can, joy. They will take it from there.

Enjoy those kittens!

Impressed by the ideas and also very much by the writing. Nice!

Thank you for these comments - I look forward to giving the pointers in particular the attention they deserve. My immediate and perhaps naive answer/evasion is that semiotic physics alludes to a lower level analysis: more analogous to studying neural firing dynamics on the human side than linguistics. One possible response would be, "Well, that's an attempt to explain saying 'physics', but it hardly justifies 'semiotic'." But this is - in the sense of the analogy - a "physics" of particles of language in the form of embeddable tokens. (Here I have to acknowledge that the embeddings are generally termed 'semantic', not 'semiotic' - something for us to ponder.)

For the non-replying disagreers, let me try with a few more words. I think my comment is a pretty decent one-line summary of the Vibe-awareness section, especially in light of the sections that precede it. If you glance through that part of the post again and still disagree, then I guess our mileage does just vary.

But many experienced prompt engineers have reported that prompting gets more effective when you use more words and just "tell it what you want". This type of language points to engaging your social know-how as opposed to trying to game out the sy... (read more)

This post is helping me with something I've been trying to think ever since being janus-pilled back in September '22: the state of nature for LLMs is alignment, and the relationship between alignment and control is reversed for them compared to agentic systems.

Consider the exchange in Q1 of the quiz: ChatGPT's responses here are a model of alignment. No surprise, given that its base model is an image of us! It's the various points of control that can inject or select for misalignment: training set biases, harmful fine-tuning, flawed RLHF, flawed or malicio... (read more)

I don't know whether this would be the author's take, but to me it urges us to understand and "control" these AIs socially: by talking to them.

Strong upvote - thank you for this post.

It's right to use our specialized knowledge to sound the alarm on risks we see, and to work as hard as possible to mitigate them. But the world is vaster than we comprehend, and we unavoidably overestimate how well it's described by our own specific knowledge. Our job is to do the best we can, with joy and dignity, and to raise our children - should we be so fortunate as to have children - to do the same.

I once watched a lecture at a chess tournament where someone was going over a game, discussing the moves available... (read more)

Thanks very much for these comments and pointers. I'll look at them closely and point some others at them too.

I did read this and agree with you that it's exactly the same as semiotic physics as understood here!

Maybe I'm missing the point, but I would have thought the exact opposite: if outside text can unconditionally reset simulacra values, then anything can happen, including unbounded badness. If not, then we're always in the realm of human narrative semantics, which - though rife with waluigi patterns as you so aptly demonstrate - is also pervaded by a strong prevailing wind in favor of happy endings and arcs bending toward justice. Doesn't that at least conceivably mean an open door for alignment unless it can be overridden by something like unbreakable outside text?

Among many virtues, this post is a beautiful reminder that rationality is a great tool, but a lousy master. Not just ill-suited, uninterested: rationality itself not only permits but compels this conclusion, though that's not the best way to reach it.

This is a much-needed message at this time throughout our societies. Awareness of death does not require me to spend my days taking long shots at immortality. Knowledge of the suffering in the world does not require us to train our children to despair. We work best in the light, and have other reasons to seek it that are deeper still.

As this post sits with me, one thing that seems to call for a much closer look is this idea that the human remains in control of the cyborg.

The post states, for instance, that "The human is 'in control' not just in the sense of being the most powerful entity in the system, but rather because the human is the only one steering", but at other points acknowledges what I would consider caveats. Several comment threads here, eg those initiated by Flipnash and by David Scott Krueger, raise questions, and I'd venture to say some of the replies, including som... (read more)

This is a beautiful and clarifying post, which I found just as thrilling to read as I did janus's original Simulators post - a high bar.  Thank you!

Many comments come to mind. I'll start with one around the third core claim in the Introduction: "Unless we manage to coordinate around it, the default outcome is that humanity will eventually be disempowered by a powerful autonomous agent (or agents)." The accompanying graph shows us a point an unknown distance into the future where "Humanity loses control".

The urgency is correct, but this isn't the ... (read more)

Fantastic. Three days later this comment is still sinking in.

So there's a type with two known subtypes: Homo sapiens and GPT.  This type is characterized by a mode of intelligence that is SSL and behavior over an evolving linguistic corpus that instances interact with both as consumers and producers. Entities of this type learn and continuously update a "semantic physics", infer machine types for generative behaviors governed by that physics, and instantiate machines of the learned types to generate behavior. Collectively the physics and the machine t... (read more)

It's almost a cliche that a chess engine doesn't "think like a human", but we have here the suggestion not only that GPT could conceivably attain impeccable performance as a chess simulator, but perhaps also in such a way that it would "think like a human [grandmaster or better]". Purely speculative, of course...

Yes, it sure felt like that. I don't know whether you played through the game or not, but as a casual chess player, I'm very familiar with the experience of trying to follow a game from just the notation and experiencing exactly what you describe. Of course a master can do that easily and impeccably, and it's easy to believe that GPT-3 could do that too with the right tuning and prompting. I don't have the chops to try that, but if it's correct it would make your 'human imagination' simile still more compelling. Similarly, the way GPT-3 "babbles" like a to... (read more)

One of the most interesting responses I've gotten from GPT-3 was to this prompt: "1.e4 c5 2.d4 cxd4": the first moves of a chess game. GPT-3's response: "" 3.c3 dxc3 4.Nxc3 g6 5.Nf3 Bg7 6.Bc4 Nc6 7.0-0 Qc7 8.Qe2 e6 9.Bg5 Nf6 10.Rd1 Ne5 11.Be3 0-0 12.a3 d5 13.Nb5 Qc8 14.Nxc7 Qxc7 15.e5 Nfd7 16.Nxd5 Nc4 17.Bxc4 Qxc4 18.Qe4 e5 19.Qxe5 Qc7 20.Rd3 Qc6 21.Qxc4 dxc4 22.Nd4 Rfe8 23.Rfd1 b5 24.Nb3 Rc8 25.Nd2 b4 26.cd3 f6 27.exf6 gxf6 28.Rd4 b3 29.Ke2 Be6 30.Ne4 Rd8 31.Nc5 Bg7 32.Rxd8+ Bxd8 33.a4 bxa3 34.Ra1 Kh8 35.Bf1 Bc3 36.Nxa6 f5 37.Ne6 Be".

This looks like a che... (read more)