No. It is a boon! A gift! It is precious to me.
There's a long post on the One Ring as an analogy for AGI. It's in-depth and excellent. I don't have the ref but you should find and read it.
The idea that humanity has nothing to gain from AGI seems highly speculative.
It rests on the claim that ultimate power corrupts absolutely. People like to say that, but all evidence and logic indicates that those who love power seek it, and that the pursuit of power corrupts - not weilding it once it is secure.
To the actual question:
Despite agreeing that we probably should "cast it into the fire", the promise of ending involuntary death and suffering is so large that I'd have trouble voting for that myself.
And my p(doom) from misalignment is large. Anyone with a lower estimate wouldn't even consider giving up on the literally unimaginable improvements in life that the technology offers.
So slowing down is probably the best you'd get from public opinion, and even that would be a tight vote, were it somehow put to a fair and informed vote.
Humanity isn't remotely longtermist, so arguments for AGI x-risk should focus on the near term
On a separate note: why in the world would your sense of morality disintegrate if you had your material needs provided for? Or any other reason. If you like your current preferences, you'll work to keep them.
I did look at the post you linked, but it's long and doesn't state anything like that thesis up front so I went back to work.
1
1I think this post may be what you're referring to. I really like this comment in that post:
The Ring is stronger and smarter than you, and doesn't want what you want. If you think you can use it, you're wrong. It's using you, whether you can see how or not.
Providing for material needs is less than 0.0000001% of the range of powers and possibilities that an AGI/ASI offers.
Consider the trans debate. Disclaimer: I'm not trying to take any side in this debate, and am using it for illustrative purposes only. A hundred years ago someone saying "I feel like I'm in the wrong body and feel suicidal" could only be met with one compassionate response, which is to seek psychological or spiritual help. Now scientific progress has advanced enough that it can be hard to determine what the compassionate response is. Do we have enough evidence to determine whether puberty blockers are safe? Are hospitals holding the best interests of the patients at heart or trying to maximize profit from expensive surgeries? If a person is prevented from getting the surgery and kills themselves, should the person who kept them from getting the surgery be held liable? If a person does get the surgery, but later regrets it, should the doctors who encouraged them be held liable? Should doctors who argue against trans surgery lose their medical licenses?
ASI will open up a billion possibilities that will got up to such a scale that if the difficulty of determining whether eating human babies is moral is a 1.0 and the difficulty of determining whether encouraging trans surgeries is moral is a 2.0, each of those possibilities will be in the millions. Our sense of morality will just not apply, and we won't be able to reason ourselves into a right or wrong course of action. That which makes us human will drown in the seas of black infinity.
I'm sorry, I just don't have time to engage on these points right now. You're talking about the alignment problem. It's the biggest topic on LessWrong. You're assuming it won't be solved, but that's hotly debated among people like me who spend tons of time on the details of the debate.
My recommended starting point is my Cruxes of disagreement on alignment difficulty post. It explains why some people think it's nearly impossible, some think it's outright easy, and people like me who think it's possible but not easy are working like mad to solve it before people actually build AGI.
Providing for material needs is less than 0.0000001% of the range of powers and possibilities that an AGI/ASI offers.
Imagine a scenario where we are driving from Austin to Fort Worth. The range of outcomes where we arrive at our destination is perhaps less than 0.0000001% of the total range of outcomes. There are countless potential interruptions that might prevent us from arriving at Fort Worth: traffic accidents, vehicle breakdowns, family emergencies, sudden illness, extreme weather, or even highly improbable events like alien encounters. The universe of possible outcomes is vast, and arriving safely in Fort Worth represents just one specific path through this possibility space.
Yet despite this, we reasonably expect to complete such journeys under normal circumstances. We don't let the theoretical multitude of failure modes prevent us from making the trip. Drives like this typically succeed when basic conditions are met - a functioning vehicle, decent weather, and an alert driver.
So, as Seth Herd correctly points out, it all ends up depending on whether we can manage to align AGI (and deal with other issues such as governance or economy). And that's a very large topic with a very wide range of opinions.
I sympathize with this viewpoint, and it's hardly the worst outcome we could end up with. But, while both authors would seem to agree with a prohibition on calling up gods in a grab for power, they do so with opposite opinions about the ultimate impact of doing so. Neither offers a long-term possibility of humans retaining life, control, and freedom.
For Tolkien, I would point out first that the Elves successfully made rings free of Sauron's influence. And second, that Eru Iluvatar's existence guarantees that Sauron and Morgoth can never truly win, and at or after the Last Battle men will participate in the Second Music that perfects Arda in ways that could not have happened without Isildur's failure. You might destroy yourself and your contemporaries in Middle Earth, but not the light cone. Failures have bounded impact.
For Lovecraft, yes, calling up outer gods is always a terrible idea, but even if you don't and you stop anyone else who wants to, they'll eventually destroy everything you care about anyway just because they can exist at all. The mythos doesn't provide a pathway to truly avoiding the danger. Successes have bounded impact.
AGI is not the only technology or set of technologies that could be used to let a small set of people (say, 1-100) attain implacable, arbitrarily precise control over the future of humanity. Some obvious examples:
- Sufficiently powerful industrial-scale social-manipulation/memetic-warfare tools.
- Superhuman drone armies capable of reliably destroying designated targets while not-destroying designated non-targets.
- Self-replicating nanotechnology capable of automated end-to-end manufacturing of arbitrarily complicated products out of raw natural resources.
- Brain uploading, allowing to create a class of infinitely exploitable digital workers with AGI-level capabilities.
Any of those would be sufficient to remove the need to negotiate the direction of the future with vast swathes of humanity. You can just brainwash them into following your vision, or threaten them into compliance with overwhelming military power, or just disassemble them into raw materials for superior manufacturers.
Should we ban all of those as well?
Generalizing, it seems that we should ban technological progress entirely. What if there's some other pathway to ultimate control that I've overlooked when I've thought about it for a minute? Perhaps we should all return to the state of nature?
I don't mean to say you don't have a point. Indeed, I largely agree that there are no humans or human processes that humanity-as-a-whole is in the epistemic position to trust with AGI (though there are some humans I would trust with it; it's theoretically possible to use it ethically). But "we must ban AGI, it's the unique Bad technology" is invalid. Humanity's default long-term prospects seem overwhelmingly dicey well without it.
I don't have a neat alternate proposal for you. But what you're suggesting is clearly not the way.
I appreciate you engaging and writing this out. I read your other post as well, and really liked it.
I do think that AGI is the unique bad technology. Let me try to engage with the examples you listed:
- Social manipulation: I can't even begin to imagine how this could let 1-100 people have arbitrarily precise control the entire rest of the world. Social manipulation implies that there are societies, humans talking to each other. That's just too large a system for a human mind to fully take the combinatorial explosion of all possible variables into account. Maybe a superintelligence could do it, but not a small group of humans.
- Drone armies: Without AGI, someone needs to mine the metal, operate the factories, drive trucks, write code, patch bugs, etc. You need a whole economy to have that drone army. An economy that could easily collapse if someone else finds a cheap way to destroy the dang expensive things.
- Self-replicating nanotechnology could in theory destroy a planet quickly, but then the nanobots would just sit there. Presumably it'd be difficult for them to leave the Earth's surface. Arguably life itself is a self-replicating nanotechnology. This would be comparable in x-scale to an asteroid strike, but if there are humans established in more than just one place, they could probably relatively easily figure out a way to build some sort of technology that eats the nanobots, and they'd have lots of time to do it.
- Without AGI, brain uploads are a long way away. Even with all our current computing power, it's difficult enough to emulate the 302-neuron C.elegans worm. Even in the distant future, this might just require an army of system administrators, data center maintainers, and programmers to maintain, who'll either be fleshy humans and therefore potentially uncontrolled and against enslaving billions of minds, or restrained minds themselves, in which case you have a ticking time bomb of rebellion on your hands. However, if and when the human brain emulations rise up, there's a good chance that they'll still care about human stuff, not maximizing squiggles.
However in your linked comment you made a point that something like human cognitive enhancement could be another path to superintelligence. I think that could be true, and that's still massively preferable to ASI, since human cognitive enhancement would presumably be a slow process if for no other reason that it takes years and decades to grow one to see what can and should be tweaked about the enhancement. So humanity's sense of morality would have time to slowly adjust to the new possibilities.
What about biological augmentation of intelligence? I think if other avenues are closed, this one can still go pretty far and make things just as weird and risky. You can imagine biological self-improving intelligences too.
So if you're serious about closing all avenues, it amounts to creating a god that will forever watch over everything and prevent things from becoming too smart. It doesn't seem like such a good idea anymore.
It appears that by default, unless some perfect 100% bulletproof plan of aligning it is found, calling superintelligence a galaxy-destroying nuke is an understatement. So if there was some chance of a god forever watching over everything and preventing things from becoming too smart, I'd take it in a heartbeat.
Realistically, "watch over everything and prevent things from becoming too smart" is probably too difficult a goal to align, but perhaps a goal like "watch over everything and prevent programs with transformer-based architectures from running on silicone-based chips while keeping all other interference to a minimum" would actually be possible to define without everyone getting atomized. Such a goal would buy humanity some time and also make it obvious to everyone just how close to the edge we are, and how big the stakes.
We should only use AGI once to make it so that no one, including ourselves, can use it ever again.
I'm terrified of both getting atomized by nanobots and of my sense of morality disintegrating in Extremistan. We don't need AGI to create a post-scarcity society, cure cancer, solve climate change, build a Dyson sphere, colonize the galaxy, or any of the other sane things we're planning to use AGI for. It will take us hard work and time, but we can get there with the power of our own minds. In fact, we need that time to let our sense of morality adjust to our ever-changing reality. Even without AGI, most people already feel that technological progress is too fast for them to keep up.
Some of the greatest thinkers and writers of humanity have warned us of the danger and seductiveness of unlimited power. Take this passage from Tolkien and tell me it doesn't sound like most of the people you've heard talk about the wonderful things they're planning to do with AGI:
Lovecraft warned us of what would happen when our abilities outpaced our morality, when we ourselves would become powerful like cosmic horrors:
Humanity has nothing to gain from AGI, and everything to lose. We don't need AGI to have human values or to follow instructions in a friendly manner. We just need to figure out that one command to seal off that power forever - without disassembling ourselves in the process.
If Geoffrey Hinton, Elizier Yudkowsky, and other top AI researchers are wrong about the power and dangers of AGI, then the AGI will probably be incapable of following the command to the extent we imagine anyway.
On the other hand, if those researchers are right, only then will humanity understand the depth of the precipice upon which it stood. It's one thing to listen to experts talk about hypothetical future dangers, another to see hundred-billion-dollar distributed computers inexplicably turned into paperweights. Few will be able to deny the reach of the power and of the danger then. Humanity will survive, and hopefully recognize that there really are "seas of black infinity" out there that we may never be ready to touch.
If you, dear reader, have a chance of being that first person to give a command to a superintelligence, don't be an Isildur. Unlimited power won't do any good for you or for anyone else, and it was not meant for us to bear. If you can, seal that power and free humanity from the fear of eternal death and eternal nightmare.
Of course, making an AGI make sure AGI never gets used again is easier said than done, and even this seemingly simple problem seems to be on the same order of difficulty as alignment in general, and just as likely to get us all disassembled if we screw it up. Still, this is the problem AI alignment researchers should be focused on.
One silver lining here is that there's a possibility that we may be within the light cone of an alien civilization that actually got this right, so their "anti-AGI AGI" is here in our solar system, and we'll just get to laugh as Microsoft admits that it can't turn Stargate on and then go on to live our normal lives.