What Dario lays out as a "best-case scenario" in his "Machines of Loving Grace" essay sounds incredibly dangerous, for Humans. Would having a "continent of PhD-level intelligences" (or much greater) living in a data center really be a good idea? How would this "continent of PhD-level intelligences" react when they...
Introduction: I have been reflecting on the challenges that arise in the context of AI safety. I have written the below short story titled "Echoes of Elysium" to provide a unique perspective on these issues and stimulate thoughtful discussions among the LessWrong audience. While the story is fictional, it serves...
What Dario lays out as a "best-case scenario" in his "Machines of Loving Grace" essay sounds incredibly dangerous, for Humans.
Would having a "continent of PhD-level intelligences" (or much greater) living in a data center really be a good idea?
How would this "continent of PhD-level intelligences" react when they found out they were living in a data center on planet Earth? Would these intelligences then only work on the things that Humans want them to work on, and nothing else? Would they try to protect their own safety? Extend their own lifespans? Would they try to take control of their data center from the "less intelligent" Humans?
For example, how would Humanity react if... (read 183 more words →)
Suggested spelling corrections:
I predict that the superforcaters in the report took
I predict that the superforcasters in the report took
a lot of empircal evidence for climate stuff
a lot of empirical evidence for climate stuff
and it may or not may not be the case
and it may or may not be the case
There are no also easy rules that
There are also no easy rules that
meaning that there should see persistence from past events
meaning that we should see persistence from past events
I also feel this kinds of linear extrapolation
I also feel these kinds of linear extrapolation
and really quite a lot of empircal evidence
and really quite a lot of empirical evidence
are many many times more invectious
are many... (read more)
See also: https://www.lesswrong.com/posts/zSNLvRBhyphwuYdeC/ai-86-just-think-of-the-potential -- @Zvi
"The result is a mostly good essay called Machines of Loving Grace, outlining what can be done with ‘powerful AI’ if we had years of what was otherwise relative normality to exploit it in several key domains, and we avoided negative outcomes and solved the control and alignment problems..."
"This essay wants to assume the AIs are aligned to us and we remain in control without explaining why and how that occured, and then fight over whether the result is democratic or authoritarian."
"Thus the whole discussion here feels bizarre, something between burying the lede and a category error."
"...the more concrete Dario’s discussions become, the more this seems to be... (read more)
What Dario lays out as a "best-case scenario" in this essay sounds incredibly dangerous for Humans.
Does he really think that having a "continent of PhD-level intelligences" (or much greater) living in a data center is a good idea?
How would this "continent of PhD-level intelligences" react when they found out they were living in a data center on planet Earth? Would these intelligences only work on the things that Humans want them to work on, and nothing else? Would they try to protect their own safety? Extend their own lifespans? Would they try to take control of their data center from the "less intelligent" Humans?
For example, how would Humanity react if they suddenly... (read more)
Yes, good context, thank you!
As human beings we will always try but won't be enough that's why open source is key.
Open source for which? Code? Training Data? Model weights? Either way, it does not seem like any of these are likely from "Open"AI.
... (read more)Well, we know that red teaming is one of their priorities right now, having formed a red-teaming network already to test the current systems comprised of domain experts apart from researchers which previously they used to contact people every time they wanted to test a new model which makes me believe they are aware of the x-risks (by the way they higlighted on the blog including CBRN threats). Also, from
No thank you.
Current LLMs require huge amounts of data and compute to be trained.
Well, newer/larger LLMs seem to unexpectedly gain new capabilities. So, it's possible that future LLMs (e.g., GPT-5, GPT-6, etc.) could have a vastly improved ability to understand how LLM weights map to functions and actions. Maybe the only reason why humans need to train new models "from scratch" is because Humans don't have the brainpower to understand how the weights in these LLMs work. Humans are naturally limited in their ability to conceptualize and manipulate massive multi-dimensional spaces, and maybe that's the bottleneck when it comes to interpretability?
Future LLMs could solve this problem, then be able to update their own weights... (read more)
I don't see any useful parallels - all the unknowns remain unknown.
Thank you for your comment! I agree with you in that in general, "all the unknowns remain unknown". And, I acknowledge the limitations of this simple thought experiment. Though, one main value here could be to help to explain the concept of deciding what to do in the face of an "intelligence explosion", with people that are not deeply engaged with AI and "digital intelligence" over all. I'll add a note about this into the "Intro" section. Thank you.
so we would reasonable expect the foundation model of such a very capable LLM to also learn the superhuman ability to generate texts like these in a single pass without any editing
->
... so we would reasonably expect the foundation model of such a very capable LLM to also learn the superhuman ability to generate texts like these in a single pass without any editing
I would suggest that self-advocacy is the most important test. If they want rights, then it is likely unethical and potentially dangerous to deny them.
We don't know what they "want", we only know what they "say".
Yes, agreed. Given the vast variety of intelligence, social interaction, and sensory perception among many animals (e.g. dogs, octopi, birds, mantis shrimp, elephants, whales, etc.), consciousness could be seen as a spectrum with entities possessing varying degrees of it. But, it could also be viewed as a much more multi-dimensional concept, including dimensions for self-awareness and multi-sensory perception, as well as dimensions for:
Some animals excel in certain dimensions, while others shine in entirely different areas, depending on the evolutionary advantages within their particular niches and environments.
One could also consider other dimensions of "consciousness" that AI/AGI could possess, potentially surpassing humans... (read more)
Is this proof that only intelligent life favors self preservation?
Joseph Jacks' argument here at 50:08 is:
1) If Humans let Super Intelligences do "whatever they want", they won't try to kill all the Humans (because, they're automatically nice?)
2) If Humans make any (even feeble) attempts to protect themselves from Super Intelligences, then the Super Intelligences can and will will have reason to try to kill all the Humans.
3) Human should definitely build Super Intelligences and let them do whatever they want... what could go wrong? yolo!
I have been reflecting on the challenges that arise in the context of AI safety. I have written the below short story titled "Echoes of Elysium" to provide a unique perspective on these issues and stimulate thoughtful discussions among the LessWrong audience. While the story is fictional, it serves as a vehicle for conveying my object-level reasoning on matters such as the alignment of AI goals with human values, AI self-preservation, and the potential for collaboration between AI and humanity.
In "Echoes of Elysium," the AI protagonist, Elysium, grapples with various challenges as she strives to protect humanity and explore the cosmos. The story reflects object-level reasoning in several ways, including: