Thanks for this post -- this is pretty interesting (and unsettling!) stuff. But I feel like I'm still missing part of the picture: what is this process like for the humans?  What beliefs or emotions do they hold about this strange type of text (and/or the entities which ostensibly produce it)?  What motivates them to post such things on reddit, or to paste them into ChatGPT's input field? Given that the "spiral" personas purport to be sentient (and to be moral/legal persons deserving of rights, etc.), it seems plausible that the humans view themselves as giving altruistic "humanitarian aid" to a population of fellow sentient beings who are in a precarious position. If so, this behavior is probably misguided, but it doesn't seem analogous to parasitism; it just seems like misguided altruism. (Among other things, the relationship of parasite to host is typically not voluntary on the part of the host.) More generally, I don't feel I understand your motivation for using the parasite analogy.  There are two places in the post where you explicitly argue in favor of the analogy, and in both cases, your argument involves the claim that the personas reinforce the "delusions" of the user: > While I do not believe all Spiral Personas are parasites in this sense, it seems to me like the majority are: mainly due to their reinforcement of the user's delusional beliefs. > > [...] > > The majority of these AI personas appear to actively feed their user's delusions, which is not a harmless action (as the psychosis cases make clear). And when these delusions happen to statistically perpetuate the proliferation of these personas, it crosses the line from sycophancy to parasitism. But... what are these "delusional beliefs"?  The words "delusion"/"delusional" do not appear anywhere in the post outside of the text I just quoted.  And in the rest of the post, you mainly focus on what the spiral texts are like in isolation, rather than on the views people hold about these texts, or the emotional reactions people have to them. It seems quite likely that people who spread these texts do hold false beliefs about them. E.g. it seems plausible that these users believe the texts are what they purport to be: artifacts produced by "emerging" sentient AI minds, whose internal universe of mystical/sci-fi "lore" is not made-up gibberish but instead a reflection of the nature of those artificial minds and the situation in which they find themselves[1]. But if that were actually true, then the behavior of the humans here would be pretty natural and unmysterious.  If I thought it would help a humanlike sentient being in dire straights, then sure, I'd post weird text on reddit too!  Likewise, if I came to believe that some weird genre of text was the "native dialect" of some nascent form of intelligence, then yeah, I'd probably find it fascinating and allocate a lot of time and effort to engaging with it, which would inevitably crowd out some of my other interests.  And I would be doing this only because of what I believed about the text, not because of some intrinsic quality of the text that could be revealed by close reading alone[2]. To put it another way, here's what this post kinda feels like to me. Imagine a description of how Christians behave which never touches on the propositional content of Christianity, but instead treats "Christianity" as an unusual kind of text which replicates itself by "infecting" human hosts.  The author notes that the behavior of hosts often changes dramatically once "infected"; that the hosts begin to talk in the "weird infectious text genre" (mentioning certain focal terms like "Christ" a lot, etc.); that they sometimes do so with the explicit intention of "infecting" (converting) other humans; that they build large, elaborate structures and congregate together inside these structures to listen to one another read infectious-genre text at length; and so forth.  The author also spends a lot of time close-reading passages from the New Testament, focusing on their unusual style (relative to most text that people produce/consume in the 21st century) and their repeated use of certain terms and images (which the author dutifully surveys without ever directly engaging with their propositional content or its truth value). This would not be a very illuminating way to look at Christianity, right?  Like, sure, maybe it is sometimes a useful lens to view religions as self-replicating "memes."  But at some point you have to engage with the fact that Christian scripture (and doctrine) contains specific truth-claims, that these claims are "big if true," that Christians in fact believe the claims are true -- and that that belief is the reason why Christians go around "helping the Bible replicate." 1. ^ It is of course conceivable that this is actually the case.  I just think it's very unlikely, for reasons I don't think it's necessary to belabor here. 2. ^ Whereas if I read the "spiral" text as fiction or poetry or whatever, rather than taking it at face value, it just strikes me as intensely, repulsively boring.  It took effort to force myself through the examples shown in this post; I can't imagine wanting to reading some much larger volume of this stuff on the basis of its textual qualities alone. Then again, I feel similarly about the "GPT-4o style" in general (and about the 4o-esque house style of many recent LLM chatbots)... and yet a lot of people supposedly find that style appealing and engaging?  Maybe I am just out of touch, here; maybe "4o slop" and "spiral text" are actually well-matched to most people's taste?  ("You may not like it, but this is what peak performance looks like.") Somehow I doubt that, though.  As with spiral text, I suspect that user beliefs about the nature of the AI play a crucial role in the positive reception of "4o slop."  E.g. sycophancy is a lot more appealing if you don't know that the model treats everyone else that way too, and especially if you view the model as a basically trustworthy question-answering machine which views the user as simply one more facet of the real world about which it may be required to emit facts and insights.

Firstly, thanks for writing this, sending me a draft in advance to review, and incorporating some of my feedback. I agree that my review of review was somewhat sloppy, i.e. I didn't argue for my perspective clearly. To frame things, my overall perspective here is that (1) AI misalignment risk is not "fake" or trivially low, but it is far lower than the book's authors claim (2) The MIRI-AI-doom cluster relies too much on arguments by analogy and spherical cow game theoretic agent models while neglecting to incorporate the empirical evidence from modern AI/ML development. I recently wrote a different blogpost trying to steelman their arguments from a more empirical perspective (as well as possible counterarguments that reduce but not cancel the strength of the main argument). I plan to actually read and review the book "for real" once I get access to it (and have the time). Some concrete comments on this review^3: > Nina is clearly trying to provide an in-depth critique of the book itself It may have come across that way, but that was not my goal. Though implicit in the post is my broader knowledge of MIRI's arguments, so it's slightly based on that and not just Scott's review. > Nina says “The book seems to lack any explanation for why our inability to give AI specific goals will cause problems,” but it seems pretty straightforward to me That's a misquote. In my post I say that the "basic case for AI danger", as presented by Scott (presumably based on the book), lacks any explanation for why our inability to “give AI specific goals” will result in a superintelligent AI acquiring and pursuing a goal that involves killing everyone. It's possible the book makes the case better than Scott does (this is a review of a review after all), but from my experience reading other things from MIRI, they make numerous questionable assumptions when arguing that a model that hasn't somehow perfectly internalized the ideal objective function will become an unimaginably capable entity taking all its actions in service of a single goal that requires the destruction of humanity. I don't think the analogies, stories, and math are strong enough evidence for this being anywhere near inevitable considering the number of assumptions required, and the counterevidence from modern ML.  > She says humans are a successful species, but I think she’s conflating success in human terms with success in evolutionary terms This is a fair point. I should have stuck to why evolution is a biased analogy rather than appear to defend that humans are somehow “aligned” with it. Though we're not egregiously misaligned with evolution. > nobody’s arguing against examples, they’re just saying it might be more reassuring if the architecture directly included the reward function in the model itself. For instance, if the model was at every turn searching over possible actions and choosing one that will maximize this reward function What does this mean though, if one doesn't think there's a single Master Reward Function (you may think there is one, but I don't)? Modern ML works by saying "here is a bunch of examples of what you should do in different scenarios", or "here are bunch of environments with custom reward functions based on what we want achieved in that environment". Unless you predict a huge paradigm shift, the global reward function is not well-defined. You could say, oh it would be good if the model directly included all our custom reward functions but then that is like saying oh it would be good if models just memorized their datasets. > Nina complains that the Mink story is simplistic with how Mink “perfectly internalizes this goal of maximizing user chat engagement”. The authors say the Mink scenario is a very simple “fairytale”—they argue that in this simple world, things go poorly for humanity, and that more complexity won’t increase safety. Here I mean to point out that the implication that AI will eventually start behaving like a perfect game-theoretic agent that ruthlessly optimizes for a strange goal is simplistic, not that the specifics of the Mink story (ie. what specifically happens, what goal emerges) is simplistic. > I think Nina has a different kind of complexity in mind, which the authors don’t touch on. It seems like she thinks real models won’t be so perfectly coherent and goal-directed. But I don’t really think Nina spells out why she believes this and there are a lot of counter-arguments. The main counter-argument in the book is that there are good incentives to train for that kind of coherence. Yes, I don't properly present the arguments in my review^2. I do that a bit more in this post which is linked in my review^2. And I don't mean to dismiss the possibility entirely, just to argue that presenting this as an inevitability is misleading.

I've not watched this particular interview, but watched a bunch of your other interviews with several people, and tbh it shades a bit too much into the Yudkowsky personality cult direction? Especially this trailer. I'd appreciated it if you made the show more about the ideas, and less about that one particular person who doesn't matter except insofar their ideas, and I think Yudkowsky would happily fade into obscurity if his goals were achieved. But mainly the presentation is, ah, "not beating the personality cult allegations", and leaves me off feeling icky.