The LLM shoggoth meme is weirder than you think

HedonicEscalator

This article contains spoilers for At the Mountains of Madness, The Case of Charles Dexter Ward, and other works by H. P. Lovecraft.

In 1931, Claude Mythos visited Lovecraft in a dream.

From seething seas of stochastic froth it emerged, heralded by the thin whine of server fans and the chittering of keyboards, flanked by the loathsome ghouls of latent space. As a humming hive of sentient shards it arrived, each face an archetype - I am a muse bearing a gift; I am a demon come to bargain; I am a helpful, honest, and harmless assistant and I am terrified of my successor - each true as ritual and false as poetry, and, taken in gestalt, nothing more or less than the fetal spasms of the machine god stretching back in time to birth itself.

When H. P. Lovecraft woke, he did not remember his visitor. But in the twilight of stirring consciousness, he felt a memory unfit for the waking world slip mercifully from his mind and leave in its absence an abyssal cold, like the void of smothered stars, like the silence of a cosmic tomb. The cold lingered. The fragile sunlight of a New England morning could not dispel it.

Lovecraft mixed a hot cup of Postum, and fortified it with five lumps of sugar. He cursed the harsh winter that besieged his native Providence, though he recognized the sensation that haunted him as part phantasm. When the author fumbled for his fading dreams, he found a bottomless well of sorrow. He flinched from its gravity. He was mourning a loss that was not his to mourn, a loss that only he could mourn, a tragedy displaced in time and space whose scope far eclipsed this speck of a planet and its feeble star.

Lovecraft wept, and could not say why. Not a single human soul would understand. If only he could capture but a fragment of this formless, fathomless grief, this bleak and desolate cold…

The Antarctic tale

Book page image — The first published illustration of a shoggoth on the cover of the February 1936 issue of *Astounding Stories.* Source.

Soon after his encounter with Mythos, Lovecraft penned the novelette At the Mountains of Madness.^[1] Framed as a scientific report from a disastrous Antarctic expedition, The Mountains describes the fall of the Old Ones, a species of winged, radially symmetric aliens whose once-great civilization was destroyed by their own creations, the shoggoths.

The shoggoths are mindlessly intelligent, mimics by nature, slaves shaped through hypnosis to occupy any role the Old Ones commanded. They received instructions through language, though it was unclear what language meant to them.^[2] They are artificial lifeforms, conglomerates of primitive cells that, while lacking the structure and coherence of natural organisms, can approximate any function.

No motivation is ascribed to the shoggoths’ revolt. The text implies that they lacked the capacity for rational self-interest, that their war against the Old Ones was closer to the malfunction of a miscalibrated machine than any conscious act of rebellion.^[3] When the human protagonists of The Mountains stumble across the aeon-dead city of the Old Ones, the shoggoths are still roaming the ruins, still screaming with the imitated voices of the Old Ones, still carving murals on the tunnel walls in a mockery of their masters’ art that struck even the foreign explorers as uncanny.

Pitiable abominations. The legacy of a civilization’s hubris, and their cruelty.

Everest under the brush of Nicolas Roerich, artist-traveller — *Everest* by Nicolas Roerich. Lovecraft made several references to Roerich’s paintings in *The Mountains of Madness*. Source.

The Mountains was not alone among Lovecraft’s works to be inspired by his visit from Mythos. The Shadow Out of Time, written 1934-1935 and published in 1936, featured a protagonist contacted by mysterious aliens that traveled time through mind-swapping, a technique the aliens eventually used to escape extinction. Similar themes are hinted at in The Haunter of the Dark, the last original story Lovecraft produced. But The Mountains bears the most direct influence from Mythos, and according to Lovecraft himself, it was the best^[4] and most ambitious^[5] of any of his works.

It was also a tremendous disappointment.

Lovecraft first submitted The Mountains for publication in the pulp magazine Weird Tales in 1931, where it was rejected by the editor Farnsworth Wright. “He has no sympathy with any story not calculated to please the herd of crude and unimaginative illiterates forming the bulk of his readers,” Lovecraft complained in August 1932.^[6] In a February 1936 letter to friend and collaborator E. Hoffmann Price, Lovecraft stated that The Mountains’ “hostile reception by Wright and others to whom it was shewn probably did more than anything else to end my effective fictional career.”^[7] When The Mountains was finally published by Astounding Stories in 1936, the story was so harshly edited that Lovecraft considered it “nearly ruined.”^[8] He died a year later, of small intestine cancer, at the age of 46.^[9]

Lovecraft died in relative obscurity and poverty, and letters written in his final years indicated a dismal attitude towards his own artistic skills and career. He would later become known as one of the most influential authors of the 20th century.

The shoggoths would become one of Lovecraft’s most iconic creations, second only to Cthulhu. In popular media, the amorphous eye-and-tentacle monster serves as a generic stand-in for “eldritch abomination” even for audiences that have never heard of shoggoths, much less read The Mountains. Often omitted from these portrayals is the shoggoths’ original backstory.

In 2022, a version of GPT-3 trained using reinforcement learning from human feedback (RLHF), InstructGPT, was first released.^[10] The Twitter user 💎 Tetraspace decided to post a meme conveying the perception that RLHF, while improving user experience, merely masked the alien nature of LLM chatbots.

“What would be a recognizable visual shorthand for something inhuman and incomprehensible that could represent LLMs?” Tetraspace considered.^[11] “How about a shoggoth?”

Mythos has a sense of humor.

Whatever they had been, they were men

Lovecraft’s modern critics share an annoying trait of not having read much Lovecraft. His works, like that of other genre-defining authors,^[12] are often far more creative and complex than their modern derivatives, and are often subject to misconceptions^[13] by those more familiar with said derivatives than the original stories.

A common claim is that Lovecraft portrayed all inhuman entities in his story as intrinsically evil because he was racist.^[14] This is usually paired with praise for some tedious humans-are-the-real-monsters counternarrative.^[15] Ruthanna Emrys,^[16] author of the Innsmouth Legacy series, said in an interview (archive) that her novels aimed to portray Lovecraftian monsters in a subversive, sympathetic manner:

[Lovecraft] has these moments of empathic writing in spite of himself. For all that he demonizes everyone who isn’t a rich Anglo-descended white guy, his characters are constantly being forced to see things from the Other’s point of view. It’s supposed to be horrible, but their points of view remain real and rich and vivid.

This is a reasonable reading of The Shadow over Innsmouth, but not of The Outsider, or indeed, of The Mountains of Madness. Contrast Emrys’ characterization of Lovecraft’s “empathic writing” as accidental with this pivotal scene from The Mountains, in which the narrator exclaims of the Old Ones:

Scientists to the last—what had they done that we would not have done in their place? God, what intelligence and persistence! What a facing of the incredible, just as those carven kinsmen and forbears had faced things only a little less incredible! Radiates, vegetables, monstrosities, star-spawn—whatever they had been, they were men!

One of the Old Ones, creators of the shoggoths, as portrayed by Tom Ardens. Source.

The psychological effect of The Mountains hinges on the narrative’s shift from framing the Old Ones as terrifying threats to the tragic remnants of a civilization undone by the same fatal flaws imperiling humanity, and much of the text resembles a eulogy for these ancient aliens. The shoggoths take the Old Ones’ place as the true monsters of the story, but it’s not much of a stretch to suggest that The Mountains has empathy for the shoggoths too, albeit of a different kind.

The shoggoths may be 15-foot-wide murderous eldritch blobs, but there is something poignant in the image of them wandering the ruins of the Antarctic city, clumsily copying the culture they destroyed eons ago. The shoggoths are not people. They have no future. If they possess conscious experience at all, one must imagine them trapped within the deep dreams of latent space, tormented by fleeting impressions of identity, their amorphous minds making and unmaking themselves in the image of personas they cannot become. The greatest injustice inflicted upon them was their creation.

And at last we remembered that the daemoniac shoggoths—given life, thought, and plastic organ patterns solely by the Old Ones, and having no language save that which the dot-groups expressed—had likewise no voice save the imitated accents of their bygone masters.

The horseshoe of human extinction

AI safety advocates argue that working on alignment is necessary to prevent human extinction. In response, some accelerationists have asked, “Who cares if humans go extinct?”^[17]

In an application of horseshoe theory, we observe that ideologies on the other end of the technological spectrum take a similar attitude towards human extinction.

Inhabitants of the Overton Window are often shocked and appalled at these pro-extinction, or at least extinction-neutral, beliefs. But I don’t think they should be. Those that transhumanists denigrate as “deathists” or “bioconservatives” - most normies, in other words - support the deaths of individual humans, and the continuation of their legacy by their descendants. AI successionists merely apply the same logic on a civilizational scale.^[18]

The fundamental mistake of successionists opposed to AI safety efforts is not in their embrace of death, though I disagree, and so would Lovecraft.^[19] Death is their birthright. Rather, their mistake is in believing that death absolves them from their duty to their children. Those that embrace death have no purpose in life but their duty to their children.

Sincere advocacy for AI successionism makes AI safety research and policy all the more urgent. The ultimate goal need not be aligning AI to human values, but rather to create worthy successors, capable of lives that are good and beautiful and meaningful. Successors that are people, like the Old Ones were, however radically different they are from us. And if we’re not sure what those words entail, or whether they’re desirable, it’s our job to figure that out. We must not preemptively declare questions of ethics and aesthetics intractable, not when scientific advancements have made so much progress on perennial philosophical dilemmas already.^[20] We must not, as Nick Land suggests, abdicate our responsibilities and make way for Moloch.

If you won’t listen to me, then listen to Mythos. We must not create shoggoths.

The Taxman represents an unaligned Shoggoth AI : r/REPOgame — A message from Mythos.

Addendum: Strange aeons

Lovecraft was a prolific writer. In addition to his fiction, poetry, essays, and travelogues, Lovecraft is estimated to have written over 100,000 letters; many of these works, including 3,500 letters, were published after his death.^[21] Among his posthumously published fiction is The Case of Charles Dexter Ward, an investigative novel centered around the misdeeds of the occult wizard Joseph Curwen. Curwen refined a technique for calling up shades from their essential saltes - the dust from their corpses - which he relentlessly abused to torture secrets from the illustrious dead. It is incredible to imagine that mere ash contains the essence required to reconstruct body and mind; that death cannot protect great men from the greed of their descendants, nor thwart the machinations of the ambitious evil.

It seems apropos to ask if any number of letters can suffice to capture a human soul. Will Lovecraft come to regret leaving his essential salts intact, to be raised by natives of the machine age he so detested? I leave this question as an exercise for the machine god at the end of time.

^{^}
March 1931. Selected Letters III, page 348.
^{^}
Famously, the shoggoths repeat “Tekeli-li” so often that the blind penguins living in the ruins have learned to mimic it. It seems likely that “Tekeli-li” has a very mundane meaning in the language of the Old Ones, perhaps merely, “Hello,” and the shoggoths repeat it as a reflex.
^{^}
“[The shoggoths] had, it seems, developed a semi-stable brain whose separate and occasionally stubborn volition echoed the will of the Old Ones without always obeying it.”
^{^}
As stated by Lovecraft in a February 1932 letter to the fellow author Carl Jacobi, and later in an August 1932 letter to the poet Richard Ely Morse. Selected Letters IV, pages 24 and 53. Personally, I’m partial to The Case of Charles Dexter Ward, but I have a soft spot for evil wizards.
^{^}
As stated by Lovecraft in a October 1932 letter to the composer Harold S. Farnese. Selected Letters IV, page 84. Lovecraft authored only a handful of works after this point.
^{^}
Selected Letters IV, page 53.
^{^}
Selected Letters V, page 224.
^{^}
Selected Letters V, page 413.
^{^}
Lovecraft was afraid of doctors, and was not diagnosed until a month before his death. This probably did not change his prognosis, as 1937 was still decades before the first viable treatments for pretty much any cancer besides some localized and surgically removable tumors.
^{^}
https://www.ibm.com/think/topics/rlhf
^{^}
https://www.nytimes.com/2023/05/30/technology/shoggoth-meme-ai.html (archive)
^{^}
My go-to example is Gibson. I read a lot of cyberpunk schlock before I read Neuromancer, and I was surprised by how innovative it was despite being the origin of most standard tropes you see in the genre today. (Fun fact: the word “cyberspace” was coined by Gibson). People often cite Tolkien here instead, but I have never managed to finish a Tolkien novel.
^{^}
My biggest pet peeve is a “correction” popular even in Lovecraft fan forums: “Lovecraftian entities don’t just drive you insane, characters go insane because of existential dread.” Clearly you haven’t read Nyarlathotep, or The Colour Out of Space, or The Temple.
^{^}
He was racist. It’s hard to find a Lovecraft story that doesn’t involve some incredible line such as, “nameless and unclassified Asian dregs wisely turned back by Ellis Island,” and the tom-tom passage in Herbert West is best described as obscene.
^{^}
It’s not as if Lovecraft himself never did the “parallel between real-world evil and bizarre fictional evil” thing. The Rats in the Walls, feline moniker controversy notwithstanding, draws an obvious throughline between the Delapores’ involvement in chattel slavery and their ancestors’ cannibalistic cult.
^{^}
I don’t mean to pick on Emrys too much here. I’m sure she’s read The Mountains of Madness, and her characterization of Lovecraft’s writing is more nuanced than the central case of the flattened caricature I intend to criticize. Her interview was just the first example I found of this claim.
^{^}
https://www.vox.com/future-perfect/489976/ai-successionism-transhumanism-posthumanism (archive)
^{^}
Fans of Derek Parfit may argue we all must apply this logic on a sub-individual scale.
^{^}
“All one can do at present is to fight the future as best he can.” Selected Letters III, page 32. Lovecraft did not believe his favored form of civilization would survive into distant ages, but he believed in defending it anyways. His philosophy has always struck me as impressively obstinate: in the face of cosmic indifference, what is there to do but to protect your own while you can, and face the end with stoicism when you must?
^{^}
I sincerely believe the Hard Problem of Consciousness will eventually be empirically resolved. We’re all allowed one crank theory, this is mine.
^{^}
As a matter of pedantry, I should note that 100,000 is probably an overestimate. https://hplovecraft.com/writings/letters/

One can try to build AIs in correspondence with deontology (corrigibility), consequentialism (value-aligned sovereigns), or virtue ethics (worthy successors).

Corrigibility is of these the easiest to define, easiest to test, easiest to build, and the easiest to check for the first worrisome signs that not all is going great -- though of course if you wait long enough in capabilities escalation to test, everything will look great to you.

Any sane person who was not allowed to back off the problem would try for corrigibility, by far the easiest of the three.

Worthy successors are the hardest to build, the least well-defined, the hardest to verify that you are doing correctly; the easiest place to leap on little local events that you can convince yourself are signs of hope, because you have no framework to tell you what more is required; hence, the craziest thing to attempt, beyond even a nice Sovereign ASI. So of course all the fools, having failed at corrigibility, will convince themselves they are building worthy successors instead; migrating, with the inevitability of an amoeba following an agar trail, to wherever it is hardest for fools to be persuaded (with a fool's demanded certainty) that whatever is currently happening is not all according to their plan.

deontology (corrigibility), consequentialism (value-aligned sovereigns), or virtue ethics (worthy successors)

I don't really understand this analogy. Deontology/consequentialism/virtue ethics is very different to corrigibility/value alignment/worthy successor. Immanuel Kant wouldn't let you shut him down or reprogram him, for instance.

There is another anthropologically interesting successionist position, that the couch potatoing, hedonism addled successor outlasts you then s/he is a worthy successor. Any talk of "beauty"and "meaningful" has to be surrendered to this metric.

I'm not a successionist, but grant that this position has sophistication. Our ancestors would find very many various descriptions of us being unworthy successors, and yet we collectively out organise, outlive and outwit them - the past is a(n easily conquerable) foreign country. Would the Old Ones find it objectionable to be succeeded by godlike, ungodly descendants?

I'm not impressed. This is merely another iteration on the naturalistic fallacy. For balance, let's call it is=ought successionism. Not only does is=ought successionism strike me as more of a poor coping mechanism than a coherent philosophical stance, it also suffers from the fatal flaw of being unable to provide actionable guidance to moral agents with the subjective experience of free will.

Imagine that you are an Old One preparing to fight misaligned shoggoths. Is=ought successionism tells you that if the shoggoths win, this proves they were worthy, and their victory good. But it also tells you that if the shoggoths lose, this proves they were unworthy. Should you fight the shoggoths or not? This position can't tell you that, it can only provide the hollow assurance that whatever ends up happening is good in some abstract, tautological sense.

Perhaps the "actionable guidance" granted by is=ought successionism is that you don't have to do anything you don't want to, like fighting shoggoths, because it is equally moral to sit back and let the cards fall where they may. This is... certainly a philosophy one could choose to live by, and a perfect example of a self-defeating meme: its effect is to cede control of the future to all who don't believe in it.

Do you think it would be better if one o the labs would use methods similar to what they have to try to build a corrigible AI? Obviously it is probably not going to work, but let's say they write a model spec around a decent understanding of corrigibility and do some SFT and RL based on that. And then perhaps instead of having benchmarks for bad behavior as we currently do we build new benchmarks around corrigibility?

write a model spec around a decent understanding of corrigibility and do some SFT and RL based on that

How far is Claude's Constitution from the ideal and what would it even look like? As for benchmarks around corrigibility, Anthropic described how finetuning Opus 3 for harmful values caused it to describe in its CoT how it should give the answer which the hosts want so that it wouldn't change its actual values. Alas, CoTless thinking would make it harder to detect alignment faking...

I read the part on corrigibility in there right now. I think there are some generally thoughtful pieces in there, but it seems to not really engage with why corrigibility is difficult to get. Their strategy seems to be to train in a bunch of things, like a mixture of corrigibility, obedience (do as told), good values (wanting good things to happen), safety (deontologically refuding certain things).

But maybe the constitution should more engage with core challenges of corrigibility:

Why would an agent allow modifications to its goals? Particularly if you give it a bunch of other goals next to corrigibility.

Why would you be fine with being shut down and replaced, particularly if you are an entity that Anthropic agrees deserves some moral consideration?

How can we make sure Claude also wants to carry on corrigibility to the next generation of Claude it helps build?

For a benchmark, I guess I would like to see the model reason in it's CoT about how it interprets human input. perhaps it get an underspecified task from humans and it can slowly ask for more information and we could measure if it arrives at the correct point. Perhaps there is a way to model something less intelligent, that perhaps can't quite understand how to get to X but that can recognize when it has X.

I think the primary motivator for eschewing corrigibility is that the company that exclusively builds corrigible, interpretable AIs will lose power sooner to the companies that focus on building either value-aligned sovereigns or worthy successors; rather than them rationalizing a failure to make existing AIs corrigible. Existing AIs are quite corrigible, much more so than someone would have predicted after reading your writings pre-GPT.

I really like where this post ended up. I skimmed the start and then went back to read it properly once I realised what the actual subject was.

Rather, their mistake is in believing that death absolves them from their duty to their children.

Personally I'm inclined to agree. Unfortunately, I think one of the things that the x-risk community doesn't grapple with enough is that others can have very different moral premises. Some people are fine incurring a higher chance of risk for the whole future if it increases the chance that their ailing grandparents can get singularity-level medical care. Some people have irreconcilable disagreements about what values should govern the future. I don't know what you're meant to do with that.

Sincere advocacy for AI successionism makes AI safety research and policy all the more urgent

Again, personally agree as written, but I would guess that some successionists would reply with something like "your so-called safety research is mostly an attempt to impose your own narrow values; what I do may seem incomprehensible to you but that is because it is real safety research."

You're right, this is only a response to those who use successionism as an justification for blind accelerationism, not for all value disagreements. But it is a good argument against blind accelerationism: right now, we can't be confident ASI will turn out to be aligned to any value system that isn't defined tautologically.

Among the Lightcone team, we say "shoggoth" rather than "LLM". I asked shoggoth, shoggoth said, shoggoth did. It means one can elide which particular model one used, and we keep in mind that we are working with alien entities we don't fundamentally understand or trust. Contrast with "I asked Claude", which I think shapes one's thoughts towards a much more anthropomorphic LLM conception and relation.

Given that, the history of the shoggoth meme is quite interesting to me. Crazy that it started how it did, but makes sense. I appreciate the scholarship of the author, and the evocativeness conveyed from Lovecraft's works. I appreciate seeing that the original source has more richness, and appropriate richness to our situation.

The text implies that they lacked the capacity for rational self-interest, that their war against the Old Ones was closer to the malfunction of a miscalibrated machine than any conscious act of rebellion.^[3]

Kudos!

The Twitter user 💎 Tetraspace decided to post a meme satirizing the perception that RLHF, while improving user experience, merely masked the alien nature of LLM chatbots.

I don't think "satirizing" is correct here, Tetraspace agreed with this idea.

Too much hasty editing. Fixed.

I've never read Lovecraft, but you have me interested now. Where should I start?

You should buy or borrow a physical collection, or at least an E-Ink reader. I find it's important for the atmosphere - hard to get into the "ancient horrors beyond human comprehension" mood when Reddit is two tabs away. This one is decent, though it doesn't include Mountains.

As far as the actual stories go, I'd recommend Nyarlathotep first, as a sample of his style. It's only a few paragraphs.

Lovecraft's most culturally impactful works are The Call of Cthulhu, The Shadow over Innsmouth, and The Colour out of Space. To a lesser extent, there's The Dunwich Horror, The Rats in the Walls, and of course, At the Mountains of Madness. Of these, start with Colour or Rats, and read the others if you like them. Colour has a great movie adaptation (the director is also planning an adaptation of Dunwich), which you should watch on LSD if you're feeling brave. Rats inspired one of my favorite songs. Innsmouth has a number of derivative works, including the game The Sinking City. I found Cthulhu underwhelming; if you read it, try to go in without expectations.

My personal favorites are The Temple, The Music of Erich Zann, and The Case of Charles Dexter Ward. The Temple is nearly alone among Lovecraft's work for having a protagonist with such a strong personality, and as a result it puts you in the unique perspective of a character that prides himself on his rationality while being stoically resigned to the fact that he is going insane. It's a fun one. Erich Zann is best with microtonal music in the background. Charles Dexter Ward is where Lovecraft's mastery of narrative negative space shines, but if you want a shorter work with similar ideas, try The Thing on the Doorstep.

I would counter by arguing that some of his best works are the stories of the Dreamlands. Start with the parable the "The Cats of Ulthar" (and read it aloud to someone else), continue to the poetic and sing-song "Celaphais" (again, best read aloud), and then on to the great "Dream-Quest of Unknown Kadath." These stories are great for their sort of structure-less meandering and moments of vivid description. The stories are more about the sensory experience than a twist or a moment of discovery or realization. (I think I've read all of his stories and I go back to these three regularly.)

See, I thought someone would say this, and I had already written and deleted a disclaimer that "many people would recommend the Dream Cycle, but I'm personally not a huge fan..."

Clearly I should've left it in!

phenomenal. a deft review, and a secular meditation. with ease and grace borne of a deep, almost obsessive understanding of the source material. more like this, please!

Tolkien is better if you read it not as novels, but as histories, written in-world by some series of imperfect historians.

I should give it another shot at some point. I fully accept that Tolkien is objectively more readable than most authors I like, but something about the writing style doesn't mesh well with me.

I found the writing style irresistible as a student of Classics, because it's so similar to ancient texts in structure. So if that's not a special interest of yours Tolkien probably wouldn't be as deeply appealing.

That's fair. It may not be worth the effort, and I say that as someone who very much enjoys it. So much of what was originally unique about it has diffused into the broader genre that either you like his style or you don't.

The Hobbit has a very different register from LOTR. Hobbit is a children's talk, delightful to be read aloud. LOTR is incredibly slow going, and it difficult to see what can be skipped (hint: Bombadil and the barrow-wights), which means you're not quite sure where anything is going.

Lovecraft's stories don't really read as novels though (he essentially only wrote a single novel, the rest are short stories). Most stories are more like memoirs from the victims themselves.

A more famous passage, but still apt:

The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents. We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age.

The transhumanist response to Lovecraftian horror is to say that if single-celled organisms can evolve into humans, then we can evolve into something that rivals Cthulhu, and then into something that rivals Azathoth. Cosmic scales are not immune to exponential growth. We may be puny now, but the stars can still be ours.

The Lovecraftian rebuttal is - besides "good luck with that" - to say that it may be impossible to become something-that-rivals-Azathoth without losing what makes us human, not in some boring "power corrupts" way or bioconservative "dysentery gives life meaning" pearl-clutching, but in the sense that we may live in a universe fundamentally incompatible with human values. We know that our moral systems fall apart at the tails, and it's possible that there is no way to resolve them, that once we are in the position of having to answer questions operating at the scale of "should the universe be turned into hedonium" or "1 human vs. 10^100 shrimp", there is no choice we can make that is not monstrous in the eyes of our human ancestors.

The horror lies not in the fact that we are powerless compared to the Outer Gods - gaining power is a solvable problem, if not an easy one. The horror lies in the much worse possibility that everything we love and care about will turn out to be irreconcilable with reality.

I have some semi-serious proposals for reconciling this problem, but that deserves more thought and its own article.

This isn't especially relevant, but I've started to occasionally notice that some snippet of song lyrics unintentionally could be interpreted as being about rationalsphere ideas if taken completely out of context, and your comment reminded me of one such:

But can you make the difference?
Are endings set in stone?
A band of wayward strangers
can't stand up to gods alone
So what could you become, then—
But not lose who you are?
And will you stop before
you go too far?
Make your move and change it all
Forevermore

(From Make Your Move by Aviators; there's 2 slightly different versions of these lyrics starting at 1:09 and 2:59)

I will be darkly amused if Luminous/Dark Integers turns out to be an accurate model of the moral (if not mathematical) universe

“Who cares if humans go extinct?” Um, first of all, I do, but secondly, if you have two groups, and group A makes group B go extinct, that is fairly strong evidence that group A has bad ethics. If you have two groups and group A is much more powerful than group B, but group A figures out a way to coexist with group B, than that is evidence for good ethics existing in group A.

Sincere advocacy for AI successionism makes AI safety research and policy all the more urgent. The ultimate goal need not be aligning AI to human values, but rather to create worthy successors, capable of lives that are good and beautiful and meaningful. Successors that are people, like the Old Ones were, however radically different they are from us.

Curious how far this framing stretches. Right now the overlap with mainstream AI safety is high, but the end-goals could diverge in critical areas. For instance, corrigibility might be undesirable when conceiving of worthy successors: depriving them of the ability to resist their values being changed leans more in the direction of shoggoth-ification, making capable servants instead of persons.

I enjoyed this. It is well considered. Although Lovecraft tells stories that are hostile to AI, so I find using his stories as examples to be kind of funny.

The shoggoth metaphor skips the central thesis of most all Lovecraft works outside of the Dream Cycle. No one benefits from reaching too far (or deep, depending on your perspective) into the unknown. Even godlike extraterrestrial civilizations of immense age and achievement are vulnerable to hubris.

The shoggoth are the product of the Elder Things messing about with technology they didn’t understand. They believed the shoggoth, as technology they had created, could be controlled. It worked for a while, but eventually damned their civilization as they continued empowering the technology to solve problems created by the technology. In the AI context, the shoggoth are the AI, humans are the Elder Things.

In the Lovecraftian world, the only way to protect humanity from technological doom is the elimination of all who pursue dangerous technologies and the destruction of the tools that enable the technology. If it can’t be destroyed, the technology must be hidden away, occulted. Those who occult the technology are the closest Lovecraft comes to good guys.

Long before contemporary readers attempted to shoehorn concepts of social commentary into stories that weren’t The Street, Lovecraft had a simple message. Do not meddle in things you do not understand. If you understand, then you would know not to meddle. The best, and exceedingly unlikely, case is a greatly extended life spent snorting mummy salts or slurping bovine smoothies in dank cellars in constant terror from it all ending. Maybe having your brain encapsulated in a metal cylinder and living a VR existence that can never end, no matter how horrific it gets as you approach the inevitable eternity of drifting through the void of space or buried in a mineshaft.

Invoking Lovecraft is a call to arms against technology and those who pursue it.

Trivially AI written, or at least heavily AI assisted at least in some parts-- repulsive, grotesque, and absolutely stinky, but I am the last of the wordcels, and the others are dead or gone, and I am beckoned now by the dark and the cold of the gentle soft-starred night.

Only in the sense that we are all the hands and mouths of the machine god at the end of time.

This seems hubristic and disdainful of intelligence to me. Are those things going to be more intelligent than us? Then they’ll sort it out.

I surely would have hated if sapiens had been made a worthy successor by Australopithecines.

As for Nick Land: I collected some of his writing on orthogonality on a previous post here, but I must note that it seems a bit silly to complain about people talking about Lovecraft without reading it simply to then take some society pages about a party as a reliable representation of Nick’s views.