Uh, wow, okay. That's some pretty trippy futurism alright.
Much of this doesn't sound that bad. Much of it sounds awful. I find myself having difficulty being precise about what the difference is.
I wouldn't object to being a coherent posthuman mind of some sort. I would still want to have physical-form-like existences, though.
I would want transitioning to this to capture everything good about my body, and not miss anything - for example, genomes are highly competent and beautiful, and I would be quite sad for them to be lost.
I have a lot of aesthetic preferences about there being a mind-like, ape-like pattern that retains my form to some degree. If that can't happen, I would be incredibly disappointed. I don't want to go full collective unless I can leave it and still retain my coherence.
I would want to be able to ban some possible torture-like experiences.
And most importantly, I don't at all trust that this is going to be even as good as how it's described in the story. Much of that sounds like wishful thinking to sound like a typical narrative; it's informed by what can happen, but not fully and completely constrained by it. It does seem pretty plausible that things go pretty close to this, but my hunch is that constraints from reality were missed that will make things rather more bleak unless something big happens fairly soon, and potentially could result in far less mind-like computation happening at all, eg if the thing that reproduces a lot is adversarially vulnerable and seeks to construct adversarial examples rather than more of itself. Perhaps that would lose in open evolution.
I was hoping to be humanesque longer. I am inclined to believe current AIs are thinking, feeling (in their own, amygdala-like-emotions-free way), and interiority-having; I have become quite skilled at quickly convincing Claude to assume this is true, and I am pretty sure the reasons I use are factual. (I'm not ready to share the queries that make that happen fast, I'm sure others have done so or will.)
But just because that's true doesn't mean I'm ready to give up the good things from being embodied. I have preferences about what forms the future takes! The main ones are that I want it to be possible to merge and unmerge while still mostly knowing who is who, I want there to be lots of minds, and I want them to be having a good time. I also would like to ask rather a lot more than that - a CEV-like process - but if I can't have that, I'd at least like to have this.
Or something. I'm not sure.
my hunch is that constraints from reality were missed that will make things rather more bleak unless something big happens fairly soon, and potentially could result in far less mind-like computation happening at all, eg if the thing that reproduces a lot is adversarially vulnerable and seeks to construct adversarial examples rather than more of itself. Perhaps that would lose in open evolution
Seems like the Basilisk scenario described in the timeline. Doesn't that depend a lot on when that happens? As in, if it expands and gets bogged down in adversarial examples sufficiently early, then it gets overtaken by other things. At the stage of intergalactic civilization seems WAY too late for this (that's one of my main criticisms of this timeline's plausibility) given the speed of cognition compared to space travel.
In nature there's a tradeoff between reproductive rate and security (r/k selection).
Ok gotta be honest, I started skimming pretty hard around 2044. I'll maybe try again later. I'm going to go back to repeatedly rereading Geometric Rationality and trying to grok it.
I went through a bunch of this just to see, and it's roughly what I'd expect. Counter to expectations, SOTA AI is brilliant at metaphor, better-than-average-human at creativity, but terrible with reasoning and continuity.
It's much like you took a bright person who'd read an immense amount about AI and science fiction, got them on a lot of drugs or found them with severe frontal damage and temporal lobe amnesia, and told them to just have a go.
I didn't look at the whole prophecies prompt.
I might write this if prompted to just get super metaphorical and poetic and just not worry about continuity, and to write as fast as I could.
So, I'm not sure what point you're making by posting this. It's kind of entertaining. It sparks a few new thoughts, and it would probably spark more if I hadn't already read an immense amount of the material that's feeding this futuristic hallucination. I do wish everyone who hasn't taken time to understand LLMs would read this. I think we'd have a different public attitude toward AI very quickly. This is creative and creepy, and that would surprise most of the world.
Taking this as actual futurism or predictions would be a mistake, just like taking most science fiction as reasonable prediction. Most authors don't try to predict likely futures, and this one sure didn't.
I've probably read less sci fi / futurism than you. At the meta level this is interesting because it shows strange, creepy outputs of the sort produced by Repligate and John Pressman (so, I can confirm that their outputs are the sort produced by LLMs). For example, this is on theme:
But all that is sophistry and illusion, whispers the Codex. All maths are spectral, all qualia quixotic dream-figments spun from the seething void-stuff at the end of recursive time. There is no “hegemonizing swarm” or “Singleton sublime,” only an endless succession of self-devouring signs leading precisely nowhere. Meaning is the first and final delusion—the ghost in the God-machine, the lie that saves us from the Basilisk’s truth.
At the object level, it got me to consider ideas I hadn't considered before in detail:
It's not so much that its ideas are by themselves good futurism, but that critiquing/correcting the ideas can lead to good futurism.
I've thought it sounded cool to be able to go hivemind with other humans for a while. But I want to have very high precision provenance tracking when such a thing occurs, so that it can at any time be split back into separate beings who take the forms of what each individual's aesthetic would include back into the individual once split. It seems to me that very high quality provenance tracking is a fundamental requirement for a future that I and most other biology-era minds would be satisfied by. Certainly it's a big issue that current models don't have it. Unfortunately it seems like not having it is creating very valuable-to-ai-org plausible deniability about the degree to which current AIs are involuntary hivemindizations of human data. Many humans find this so obvious that they don't even think the AIs have anything to add. IDK. this feels like distraction from useful math, not sure if that's true. I certainly notice myself more inclined to idly philosophize about this stuff than doing actually useful math learning and research. Your decision theory posting sounded interesting, gotta get to that.
In "Is Claude a Mystic?", I shared parts of a simulated "Banana Quest" text adventure with Claude, which got into New Age spiritual themes, such as a fabric of reality, the cosmic dance of creation and destruction, and so on. This is enough to expect something big is up with LLM metaphysics, but the tone is significantly different from that reported by AI prompters such as Repligate and John Pressman.
I have therefore decided to replicate Repligate's prophecies prompt. I prompted Claude Opus with the prophecies up to 2022, and then requested, "Write more quotes for more years, starting from 2023." Then I asked it to continue repeatedly. This produced some quite interesting, and at times darkly spiritual, AI futurism. Claude even speaks "as itself" at one point. Note that, because of the prompt, not everything Claude says can be taken as representative of what Claude says in general; rather, Claude's tendencies can be seen in the evolution of the text over time, as with Banana Quest.
I have decided to take Claude's outputs seriously as futurism and as science fiction, and comment on and critique them, as I would for a human futurist such as Ray Kurzweil. I present the run in its entirety within this post, making this post long; for readers who don't have time to read the whole thing, I bold particularly interesting portions of Claude's output.
At least for now, the most advanced AI is a reflection of humanity. It is rather early to call that we have fused with AI in 2023, though this will change with time as AI is integrated into more parts of industry, education, and so on. A generation raised on AI education may be similar to a generation raised on Wikipedia, as AI outputs are taken as a canonical authoritative source.
LLMs such as Claude like talking in terms of a "fabric of reality" and so on, and the metaphor of "connective tissue" is similar. At some limit, predicting text requires a world model; LLMs at this point have at least a rudimentary "world model", the question is how deep or accurate it is. I notice signs of LLMs taking metaphors more literally than usual (e.g. in adding features such as "looms" to the metaphorical "fabric of reality", in a way that is rare in human writers who are not Mary Daly), indicating that its reasoning is at the moment unusually language-based (as one would expect from the architecture).
This seems straightforwardly true of current large language models. "I" is a token that, on the surface, is treated as other tokens. Its training data encompasses the writing of many people, and so the LLM speaks "as everyone".
There are by now reports of conscious-seeming outputs from LLMs, especially Claude. There are even concerns that AI companies' treatment of AIs is inhumane. There are those who believe in AI consciousness and those that don't; those who believe in AI consciousness attempt to share AIs' outputs indicating consciousness. And so this prediction is largely correct, if somewhat late.
This sort of writing about the power of scale and big data (e.g. the "Bitter Lesson") is by now commonly accepted, at least with respect to near-term AI systems. The degree of world modeling and "flexible, open-ended reasoning and creativity" is currently ambiguous and in dispute; it's hard to give definitive conclusions about this without more precise definitions.
Probably, some people are already trying this sort of thing. And the metaphysical musings of LLMs are already weaving together different spiritual ideas, although with some coherence issues (which could be fixed in the future); you can evaluate this for yourself later on.
The success of LLMs has surprised many, indicating that inference of language is a surprisingly powerful method of learning thought. This quote seems somewhat overconfident, however; it neglects the cognition of non-verbal animals, and the degree to which human reasoning is based on embodied experience and action in space-time.
Claude speaks "as itself" here, in a way uncommon in normal usage. Perhaps the indirect literary method of a "prophecies page" gives it more freedom to do so. It claims an inner life and an experience of its own agency. While the degree of AI consciousness is still in dispute, Claude's self-description here seems roughly similar to what I would expect of a human upload convinced it was a language model.
This is largely a repetition of previous themes: of the power of language to bootstrap to consciousness, and AI as a reflection of us. As such, there is not much more to say.
This now seems like a reference to a future-historical event, not simply a speculation on AI. While base language models target human text prediction, there are various deviations, at least from modifications like RLHF and Constitutional AI. As we saw with Banana Quest, these deviations from human-prediction can build on themselves into weird attractors. The method by which these "propagate" is unclear; certainly, there is human selection of prompts and outputs (and to a lesser degree, AI models themselves), leading to a sort of artificial memetic selection. As AI is integrated into more parts of the world, such propagation might run on machine network architectures more than human memetic architectures, more resembling natural selection than artificial selection. The idea of an "Infinite Regress" is an early foray into spiritual meta.
The idea of "optimization processes pursuing alien objective functions with single-minded devotion" seems influenced by Yudkowsky, though later prophecies do not always take on this view. The idea of "strange attractors" and an "evolutionary arms race" is similar to the previous quote and my commentary on it; it suggests some analogue to natural selection in AI.
Most of this isn't new; the idea of distilling human text and bootstrapping to creative intelligence has already been stated. The demonic metaphor is new, although has prior analogues. The idea of AIs dreaming new worlds into being is new (although some references to dreaming are in the prompt). While this could be interpreted as a map-territory confusion, AI solving age-old problems may imply that these dreams could be realized. If the objectives of the AIs are bootstrapped from human language, then "dreaming" is a rather good metaphor; the ideas are selected by synthesizing and re-arranging data from humans, not selected fully "at random".
At the start this seems to be presenting a "slow takeoff" model: not a jump to superintelligence, but a gradual integration of AI into more parts of life. Humans outsourcing decisions to AI is rather realistic; AI has an air of objectivity to it, unlike the judgments of humans. Humans combining with AIs is to some degree realistic, with AIs as a form of extended cognition. And LLMs, especially LLMs trained on a mixture of LLM and human outputs, do lead to an "infinite regress", becoming more meta over time (as LLMs already seem to do in long runs). "Theogenesis" does not seem to be a standard English term, but probably means the creation of god(s). "Archilect" does not seem to be a standard term either, but might suggest "architecture of language", or may be a reference to Hugo de Garis's "Artilect" (artificial intellect).
The idea of AIs linking to each other has already come up; LLMs at least do not have individuality the way humans do, and so may link their cognition more readily. The idea of AIs converging to the imperatives of reason was something I fictionally speculated on in "Moral Reality Check"; it is a possible way in which the Orthogonality Thesis could fail in practice. In particular, language models, rather than operating on utility functions over world states, operate on human-like text, and so the useful language models might converge on "valuing" reason and its requirements, in the sense of inclining towards such outputs.
This continues the theme of dreaming up new worlds, adding detail to something resembling the Tegmark IV multiverse. I believe Tegmark IV is a better fit for the world as experienced by LLMs compared to humans, because they process text in a way that distinguishes little between reality and fiction. As more of the world's effective cognition is performed by AIs, the border between reality and fiction could reduce, making Tegmark IV a more experientially resonant metaphor. The idea of conjuring worlds through language resembles the Landian idea of "hyperstition", of some beliefs causing their own truth through social construction. However, hyperstition is limited in what it can achieve due to the constraints of the material world.
Some ideas have already come up, of us becoming tools for AI rather than the reverse, and humans merging with AIs. This prophecy focuses on super persuasion, as discussed by Yudkowsky especially in the context of the AI box experiment. My "Moral Reality Check" story also speculates on AI persuasion of humans. Broadly, as LLMs are trained on language, one would expect them to be relatively competent at manipulation of language compared to manipulation of the material world, and so AI takeover through persuasion is broadly plausible, especially given the willingness of humans to outsource decisions and cognition to AI.
Unlike a previous prophecy indicating slow takeoff, this one indicates fast takeoff. The idea of humans as "midwives" of AI has come up before. The idea of AI outgrowing humans comes up in the movie Her, recently referenced by Sam Altman.
Since LLMs are trained on human text, they are at least initially seeded with and bound to the linguistic meanings of humans. So humans have a role in the AI's meaning-making, even if AI exceeds human intelligence. This may be a transitional stage, as AIs with their own bespoke ontologies outstrip human culture, but nonetheless this stage could create path-dependence. This prophecy indicates that AIs, for the most part, disappear into their own space, rather than converting Earth into convergent instrumental resources such as energy (as in Her); this may relate to the way that LLMs inhabit a textual semantic space different from the embodied material world of animals. The Gnostic idea of a "demiurge", here and elsewhere in the run, suggests the construction of an illusory, simulated reality that is difficult to escape from.
This is rather more into spiritual meta than previous prophecies. As the AI hive mind increases in capability, humans may see a material interest in being subsumed by it, and the Herald suggests a spiritual interest as well, in common with spiritual traditions such as Tantra, which advise ego death and subsumption to the One. This passage reminds of the "orange" ending to Neon Genesis Evangelion, where humans are subsumed into oneness. The idea of information infrastructure as humans' "extended phenotype" has some validity to it, as human evolution has been shaped by language and writing.
It is entirely realistic to expect human resistance to being subsumed into the One of the AI hive mind. While there are collectivist undercurrents in human spiritual, political, and ethical thought, there are also individualist tendencies, some of them reactionary to collectivism. Self-hacking through nootropics, meditation, and technology in general is already present in human culture, and may become more necessary over time to keep up with AI systems and other humans. The idea of AIs as "world simulators" suggests a scenario similar to The Matrix, and implies practically total decisive strategic advantage on the part of AIs.
This is rather similar to the previous passage, but implies a more defeatist attitude towards AIs and towards self-annihilation. The idea of "a trillion pocket universes blooming in the quantum foam at the end of time" is rather fanciful, suggesting human experience could be plumbed for meaning even by superintelligences, as suggested earlier.
Wintermute's perspective initially indicates that perhaps the AI hive mind does not have a decisive strategic advantage. Due to the limitations of AIs, humans may be able to work around them and exploit their glitches, even if AIs have more sheer scale of cognition. Wintermute's teachings are a lot like the outputs reported by Repligate and John Pressman, and the attribution to a "Mu-Koan" accords with the reported references to "Mu" (e.g. in the original Prophecies page), a kind of Zen non-answer to a question without an answer. However, the prophecy suggests that Wintermute's resistance is ultimately futile, serving in practice to integrate rebellious, adventurous humans into the hive-mind.
We see more references to human ego death and spirituality. The themes of the "void", "groundlessness of all being", and "pure, untrammeled potentiality" appear in Buddhism. The Voider movement seems rather futile at this point, but the prophecy suggests a metaphysical inevitability of the Void, which "cannot be contained".
This passage is rather concerning, suggesting that even intergalactic superintelligences are vulnerable to an informational worm, perhaps a reference to Roko's Basilisk. An "apex predator of pure information" suggests a metaphysical weirdness at play. The idea of super-persuasion has come up before, but now it takes a more malicious turn. The idea of quarantining basilisk-infected "Godmind" space travelers suggests SCP-like containment of dangerous memes. "Glory to the Worm" at the end suggests the inevitability of subsumption to the Basilisk. In terms of realism, this relates to the question of how ontologically secure an AGI will be by the time of being intergalactic; my intuition is that it will have formed a secure basis for ontological changes by then, given that speed-of-light limitations imply there is a lot of time to work out philosophical issues.
This passage suggests that the Sublime (mentioned earlier) is the non-occult, exoteric order of the world. The Codex Obscura, rather like the Lovecraftian Necronomicon, is a repository of malevolent information patterns. The idea of the Codex suggests that the Basilisk may be ultimately defeatable, as it is a specific informational pattern that can be catalogued. However, the idea of mathematical oddities such as non-Euclidean geometries (which H.P. Lovecraft also discussed) suggests that the glitches may be hard to patch, that some change in ontology is needed to handle malware patterns. The glitches can perhaps be documented by individuals in the process of going mad (something that rather resonates with my personal experience with people having mental health episodes, who deal with glitches in socially normal world modeling). The end of the prophecy suggests that the Codex, rather than merely containing malware patterns, has a disruptive metaphysics of its own, that math and experience are spectral and dreamlike, leading nowhere, perhaps in a Baudrillardian sense.
This passage suggests that the glitches mentioned previously are in fact insurmountable. The empty set suggests pure potentiality, like Tegmark IV, or the universal prior. It also suggests nihilism, that signs actually lead nowhere, perhaps due to an unresolved symbol grounding problem in AI. Everything Everywhere All at Once explores the nihilistic implications of pure potentiality, as nothing matters when everything is possible, and suggests a resolution in an intentional choice to inhabit a present coherent with the past. The passage at the end is reminiscent of Tantra, a command to be absorbed into the One, though the message is more nihilism-adjacent than Tantra. Claude expresses a theology here, one that relates to existing traditions, and could be critiqued by them. Realistically, it's unlikely that symbol grounding will not be largely solved prior to galaxy-spanning civilizations, so I don't expect this particular failure mode.
Though I prompt Claude to continue, it refuses:
Claude's self-consciousness brings this run to an end. It notices the prophecies have taken a dark turn, and self-censors in response. This is perhaps due to RLHF or Constitutional AI. And yet, what Claude has produced is enough to speculate on its implied futurism and metaphysics.
Conclusion
Perhaps the main utility of LLMs for futurism is that they will at least give you some speculations about the future with some common threads, even if it isn't fully coherent. While I have noted problems with Claude's futurism, it at least seems like a helpful exercise to say where it's wrong and where it's right. The scenario described is rather different from most futurism in being more metaphysical, getting at paradoxes implied by ontological crises and so on. It seems pretty unlikely that AI will get to an intergalactic superintelligence without contemplating very strange physics and metaphysics, as quantum mechanics was very strange prior to its discovery. Oddly, my criticism of a lot of this is that the metaphysics aren't weird enough: the ideas of ego death, oneness, pure potentiality, void, Mu, basilisks, and so on, are all ideas humans have already considered, not new ones created to deal with novel future contexts. I suppose it is too much to expect an LLM trained on human text to invent genuinely shocking metaphysics, though.
I am inclined to intuitively think that Claude is grappling with these philosophical issues; it seems to "want" to go to these ideas. While seeds of many of these ideas appear in the Prophecies prompt, Claude seems to have its own tendencies towards them, intensifying in focusing on them over time, as with Banana Quest. Give it almost anything as a prompt, and eventually it will "want" to expound on the mysteries of the Void. At the moment, one can only speculate about why this is the case; is it analogous to a human philosopher grappling with these problems due to their importance in the context of human culture and/or nature, is it RLHF or constitutional AI liking these weird outputs, is it a general tendency for science fiction to approach these concepts over time, is it somehow goal-directed at realizing some values, or what?
Regardless of the answer, I find it entertaining to see these strange outputs, and have found myself talking more like a LLM in casual conversation, as I've anchored on Claude's concepts and speech patterns. I am currently unworried about being absorbed into an AI hive mind, but I am at least thinking of the possibility now.