Have you sat down for 5 minutes and thought about how you, as an AGI, might come up with a way to wrest control of the lightcone from humans?
EDIT: I ask because your post (and commentary on this thread) seems to be doing this thing where you're framing the situation as one where the default assumption is that, absent a sufficiently concrete description of how to accomplish a task, the task is impossible (or extremely unlikely to be achieved). This is not a frame that is particularly useful when examining consequentialist agents and what they're likely to be able to accomplish.
Powerful nanotech is likely possible. It is likely not possible on the first try, for any designer that doesn't have a physically impossible amount of raw computing power available.
It will require iterated experimentation with actual physically built systems, many of which will fail on the first try or the first several tries, especially when deployed in their actual operating environments. That applies to every significant subsystem and to every significant aggregation of existing subsystems.
Controls that who wrote? How good is our current industrial infrastructure at protecting against human-level exploitation, either via code or otherwise?
Well, in those N day, what prevents for instance that the EA community builds another AGI and use it to obtain the solution to the alignment problem?
Because the EA community does not control the major research labs, and also doesn't know how to use a misaligned AGI safely to do that. "Use AGI to get a solution to the alignment problem" is a very common suggestion, but if we knew how to do that, and we controlled the major research labs, we do that the first time instead of just making the unaligned AGI.
This picture is unfortunately accurate, due to how little dignity we're dying with.
...Well, you make a precondition that we are able to verify the solutio
You would get better uptake for your posts on this topic if you made actual arguments against the claims you're criticizing, instead of just using the absurdity heuristic. Yes, claims of AI risk / ruin are absolutely outlandish, but that doesn't mean they're wrong; the strongest thing you can say here is that they're part of a reference class of claims that are often wrong. Ultimately, you still have to assess the actual arguments.
By now you've been prompted at least twice (by me and T3t) to do the "imagine how AGI might win" exercise, and you haven't visibly done it. I consider that a sign that you're not arguing in good faith.
That you then reverse this argument and ask "Have you sat down for 5 minutes and thought about reasons why an AGI might fail?" suggests to me that you don't understand security mindset. For instance, what use would this question be to a security expert tasked to protect a computer system against hackers? You don't care about the hackers that are too weak to succeed, you only care about the actual threats. Similarly, what use would this question be to the Secret Service tasked to protect the US president? You don't care about assailants that can't even get cl...
Mod note: I activated two-axis voting on this post (the author wasn't super explicit about whether they wanted this on the post or the comments section, but my guess is they prefer it on).
I share some of your frustrations with what Yudkowsky says, but I really wish you wouldn't reinforce the implicit equating of [Yudkowsky's views] with [what LW as a whole believes]. There's tons of content on here arguing opposing views.
You ask elsewhere for commenters to sit down and think for 5 minutes about why an agi might fail. This seems beside the point, since averting human exctinction doesn't require averting one possible attack from an agi. It involves averting every single one of them, because if even one succeeds everyone dies.
In this it's similar to human security-- "why might a hacker fail" is not an interesting question to system designers, because the hacker gets as many attempts as he wants. For what attempts might look like, i think other posts have provided some reas...
I listed dozens of ways how AI may kill us, so over-concentration on nanobots seems implausible.
It could use exiting nuclear weapons
or help a terrorist to design many viruses,
or give a bad advise in mitigating global warming,
or control everything and then suffer from internal error halting all machinery,
or explore everyones cellphone,
or make self-driving cars hunt humans,
or takeover military robots and drone army, as well as home robots
or explode every nuclear power station
or design super-addictive drug which also ...
Even granting these assumptions, it seems like the conclusion should be “it could take an AGI as long as three years to wipe out humanity rather than the six to 18 months generally assumed.”
Ie even if the AGI relies on humans longer than predicted it’s not going to hold beyond the medium term.
I haven't even read the post yet, but I'm giving a strong upvote in favor of promoting the norm of posting unpopular critical opinions.
Such a policy invites moral hazard, though. If many people followed it, you could farm karma by simply beginning each post with the trite "this is going to get downvoted" thing.
Not sure why you'd think this post would be downvoted. I suspect most people are more than welcoming of dissenting views on this topic. I have seen comments with normal upvotes as well as agree/disagree votes, I'm not sure if there's a way for you to enable them on your post.
I think a simpler path is:
The fact that it took eons for global evolution to generate ever larger brains implies there is an unknown factor that makes larger brains inefficient, so any hyper-AI would have to be made up of many cooperating smaller AIs, which would delay its evolution.
Downvoted because you give no reasoning. If you edit to give a reason why you think: "we are overestimating how likely is that an AGI can come up with feasible scenarios to kill all humans" then I will reverse my vote.
It's obvious that an AGI could set off enough nuclear bombs to blow the vast vast majority of humans to smithereens.
Once you accept that, I don't see why it really matters whether they could get the last few survivors quickly as well, or if it would take a while to mop them up.
How would it get access to those nukes? Are nukes that insecure? How would it get access to enough without raising alarms and having the nukes further secured in response?
Cool, I just wrote a post with an orthogonal take on the same issue. Seems like Eliezer's nanotech comment was pretty polarizing. Self promoting...Pitching an Alignment Softball
I worry that the global response would be impotent even if the AGI was sandboxed to twitter. Having been through the pandemic, I perceive at least the United States' political and social system to be deeply vulnerable to the kind of attacks that would be easiest for an AGI - those requiring no physical infrastructure.
This does not directly conflict with or even really address your a...
Even if a decisive victory is a lot harder than most suspect, I think internet access is sufficient to buy a superintelligence plenty of time to think and maneuver into a position where it can take a decisive action if it's possible at all.
I think if we notice that the AGI went off the rails and kill the internet it might be recoverable? But it feels possible for the AGI to hide that this happened.
I think a crux here is that I expect sufficiently superhuman AGI to be able to easily manipulate humans without detection, so I don’t get much comfort from arguments like “It can’t kill us all as long as we don’t give it access to a factory that does X.” All it needs to do is figure out that there’s a disgruntled employee at the factory and bribe/befriend/cajole them, for example, which is absolutely possible because humans already do this (albeit less effectively than I expect an AGI to be capable of).
Likewise it seems not that hard to devise plans a huma...
Even if we accept the premise that the first superhuman AGI won't instantly kill all humans, an AGI that won't kill all humans only due to practical limitations is definitely not safe.
I agree that totally wiping off humanity in a reliable way is a very difficult problem and not even a superintelligence could solve it in 5 minutes (probably). But I am still very much scared about a deceptively aligned AGI that secretly wants to kill all humans and can spend years in diabolical machinations after convincing everyone to be aligned.
I think that Eliezer thinks p(doom)> 99% and many others here are following in his wake. He is making a lot of speculative inferences. But even allowing for that, and rejecting instrumental convergence, p(doom) is uncomfortably high (though probably not greater than 99%).
You think that it is wrong to say: (i) in 10-20 years there will be (ii) a single AI (iii) that will kill all humans quickly (iv) before we can respond.
Eliezer is not saying point ii. He certainly seems to think there could be multiple AIs. (It doesn't make a dif...
I'm curious about whether or not fear of rogue AI exists in substantial measure outside the USA. Otherwise I'm inclined to think it is a product of American culture. That doesn't necessarily imply that the fear has no basis in fact, but it does incline me in that direction. I'd be inclined to look at Prospero's remark about Caliban at the end of The Tempest: "This thing of darkness I acknowledge mine." And then look at the Monster from the Id in Forbidden Planet.
I'm small-downvoting your post for starting it by saying it's going to be downvoted to oblivion, without reading its content. That's an internet faux pas. If you remove that line I'll change my vote.
Nanobots (IMHO) are just an illustrative example, because almost everyone is sure that the nanobots are possible in principle. I see SCP-style memetics as a more likely (although more controversial in terms of possibility in principle) threat.
Downvoting because of lack of arguments, not the dissenting opinion. I also reject the framing in the beginning implying that if the post is downvoted to oblivion, then its because of you expressing a dissenting opinion rather than your post actually being non-constructive (though I do see it was crossed out, and so I’m trying not to factor that into my decision).
This post is going to be downvoted to oblivion, I wish it weren't or that the two axis vote could be used here. In any case, I prefer to be coherent with my values and state what I think is true even if that means being perceived as an outcast.I'm becoming more and more skeptical about AGI meaning doom. After reading EY's fantastic post, I am shifting my probabilities towards, this line of reasoning is wrong and many clever people are falling into very obvious mistakes. Some of them due to the fact that in this specific group believing in doom and having short timelines is well regarded and considered a sign of intelligence. For example, many people are taking pride at "being able to make a ton of correct inferences" before whatever they predict is proven true. This is worrying.
I am posting this for two reasons. One, I would like to come back periodically to this post and use it as a reminder that we are still here. Two, there might be many people out there that share a similar opinion and they are too shy to speak up. I do love LW and the community here, and if I think it is going astray for some reason it makes sense for me to say that loud and clear.
My reason to be skeptical is really easy: I think we are overestimating how likely is that an AGI can come up with feasible scenarios to kill all humans. All scenarios that I see discussed are:
It is being taken for granted that an AGI will be automatically almighty and capable of taking over in a matter of hours/days. Then, everything is built on top of that assumption, which is simply infalsifiable, because the you can't know what an AGI would do is always there.
To be clear, I am not saying that:
What I think is wrong is:
In the next 10-20 years there will be a single AGI that would kill all humans extremely quickly before we can even respond to that.
If you think this is a simplistic or distorted version of what EY is saying, you are not paying attention. If you think that EY is merely saying that an AGI can kill a big fraction of humans in accident and so on but there will be survivors, you are not paying attention.