TL;DR: you can still just sign this statement if you agree with it. It still matters, and you can clarify your position in a statement of support (600 characters) next to your name, and you can clarify your actual full position on LW and/or elsewhere. Regardless of X, you can...
[Context: This post is aimed at all readers[1] who broadly agree that the current race toward superintelligence is bad, that stopping would be good, and that the technical pathways to a solution are too unpromising and hard to coordinate on to justify going ahead.] TL;DR: We address the objections made...
[Co-written by Mateusz Bagiński and Samuel Buteau (Ishual)] TL;DR Many X-risk-concerned people who join AI capabilities labs with the intent to contribute to existential safety think that the labs are currently engaging in a race that is unacceptably likely to lead to human disempowerment and/or extinction, and would prefer an...
TL;DR: you can still just sign this statement if you agree with it. It still matters, and you can clarify your position in a statement of support (600 characters) next to your name, and you can clarify your actual full position on LW and/or elsewhere.
X= whether you agree with me that safety researchers have a duty to take a public stance. If you agree with the statement and you are working at a lab where some of your coworkers signed, please consider that signing make it less personally costly for them to have signed.
X= whether you agreed with signing a... (read 758 more words →)
... (read 654 more words →)
- A convincing scenario cannot involve any bioweapons. Normal people just don’t know how vulnerable the human machine is. They think pandemics are just something that happens every 5-20 years, and don’t think about it besides that. They don’t think about the human body as a nano factory that’s vulnerable to targetted nano-attacks.
- A scenario that passes the mom test will also not include any drones. Yes, even though drones are currently used in warfare. Drones are the future. Drones are toys. Futuristic toys don’t sound like a realistic threat.
- A mom test scenario also shouldn’t involve any hacking. Regular people have no idea how insecure computer systems are. It’s basically safe to do online
idea to make a thing that happened believable:
- stuxnet has a wikipedia page that is easy to point to. Pointing to things as a reflex sometimes works.
- there are perhaps some videos/books that are aimed at the general public, are somewhat entertaining for the general public, and have a visible view count of >1M.
- Maybe it was covered by one of the news channels that the general public person acknowledges as existing and a valid source of real things.
I think for the "scary hacks" category, it is worth coming up with 3-5 very different illustrative cases, and looking for ways to connect them to other things the person (likely) thinks are real.
I... (read more)
[Context: This post is aimed at all readers[1] who broadly agree that the current race toward superintelligence is bad, that stopping would be good, and that the technical pathways to a solution are too unpromising and hard to coordinate on to justify going ahead.]
TL;DR: We address the objections made to a statement supporting a ban on superintelligence by people who agree that a ban on superintelligence would be desirable.
Quoting Lucius Bushnaq:
I support some form of global ban or pause on AGI/ASI development. I think the current AI R&D regime is completely insane, and if it continues as it is, we will probably create an unaligned superintelligence that kills everyone.
We have been circulating a statement[2] expressing ~this view,... (read 3895 more words →)
I think that signing sufficiently clear open letters/similar things seems sufficient to count as "taking a public stance".
[Here is a first rough attempt at expressing an idea:]
I don't think pushing for an AI pause now is what most people have in mind (definitely it doesn't match what I had in mind when writing the post, keeping in mind that the post isn't about what kind of public stance it would be effective for experts (inside and outside the frontier labs) to take). Instead, what matters imo is to have a legible thing (eg open letter/statement) that says clearly that you think a coordination to stop the current race dynamic would be good... (read 417 more words →)
This sounds to me like there would actually be specific opportunities to express some of your true beliefs that you wouldn't worry would cost you a lot (and some other opportunities where you would worry and not do them). Would you agree with that?
I agree that people should clearly state that they think there's a catastrophic risk, but I disagree that people should clearly state that they think we should pause.
If we premise (as this post does) on the fact that the person we are talking about actually believes that an international ban would be a great improvement over the current mad AI race, then the above quote seems wrong to me.
I agree that experts should not pretend like they have more authority than they do in judging whether we should pause. But they could still say 1) that the race is insane, 2) that an international ban seems like a great improvement, 3) that... (read more)
I probably should have said "norm execution" (ie follow the norm). This might just be a cultural gap, but I think norm enforcement/execution/implementation works in many ways that are not threats. For instance, there is pizza at a conference. there is a norm that you shouldn't take all the pizza if there is a big line behind you. some people break this norm. what happens? do they get threatened? no! they just get dirty looks and people talking behind their backs. maybe they get the reputation as the "pizza taker". In fact, nobody necessarily told them before this happened that taking all the pizza would break the norm.
I think there is a... (read more)
(optional: my other comment is more important imo)
I'm not concerned about someone being fired for this kind of thing, that would be pretty unwise on the labs' part as you risk creating a martyr
I think you ascribe too much competence/foresight/focus/care to the labs. I'd be willing to bet that multiple (safety?) people have been fired from labs in a way that would make the lab look pretty bad. Labs make tactical mistakes sometimes. Wasn't there a thing at OpenAI for instance (lol)? Of course it is possible(/probable?) that they would not fire in a given case due to sufficient "wisdom", but we should not assign an extreme likelihood to that.
Rather, I'm concerned about eg senior figures thinking worse of safety researchers as a whole because it causes a PR headache, eg viewing them as radical troublemakers, and this making theories of impact around influencing specific senior decision makers harder (and I'm more optimistic about those, personally)
Thank you Neel for stating this explicitly. I think this is very valuable information. This matches what some of my friends told me privately also. I would appreciate it a lot if you could give a rough estimate of your confidence that this would happen (ideally some probability/percentage). Additionally, I would appreciate if you could say whether you'd expect such a consequence to be legible/visible or illegible (once it had happened). Finally, are there legible reasons you could share for your estimated credence that this would happen?
(to be clear: I am sad that you are operating under such conditions. I consider this evidence against expecting meaningful impact from the inside at your lab.)
I conclude from this that you really do see this post as a threat (also you admitted there is no threat in your first comment so this comment now seems contradictory/bad-faith).
some thoughts:
- this isn't a threat by proxy and isn't a threat (but if it were a tbp then it would be a t sure)
- I am in the "others" group. I implement the norm I endorse in the post, and I am not threatening you. I don't want to sound dismissive but you are not giving me a lot to work with here, and it sounds to me like either 1) you have a vague model of what a threat is... (read more)
[Co-written by Mateusz Bagiński and Samuel Buteau (Ishual)]
Many X-risk-concerned people who join AI capabilities labs with the intent to contribute to existential safety think that the labs are currently engaging in a race that is unacceptably likely to lead to human disempowerment and/or extinction, and would prefer an AGI ban[1] over the current path. This post makes the case that such people should speak out publicly[2] against the current AI R&D regime and in favor of an AGI ban[3]. They should explicitly communicate that a saner world would coordinate not to build existentially dangerous intelligences, at least until we know how to do it in a principled, safe way. They could choose to maintain their... (read 2212 more words →)
I think your advice is a quite clear articulation of the strategy behind the CAIS statement. I think this is a great and difficult strategy to pursue. Indeed, I have been circulating https://superintelligence-statement.org/ for some time now (secretly), and I think this statement first ensured it would get some of the major signatories before even making its way to me.
I think the OP is both talking about object-level reasons people gave, and it also shows that there is another strategy one might pursue which is to go around and convince people to actually support an existing statement publicly once they support it privately.