LESSWRONG
LW

All of Epimetheus's Comments + Replies

The ground of optimization
Epimetheus10

Is a system that optimizes for destruction an optimizing system?

Reply
2Alex Flint
A bomb would not be an optimizing system, because the target space is not small compared to the basin of attraction. An AI that systematically dismantles things would be an optimizing system if for no other reason than that the AI systematically preserves its own integrity.
What does it take to defend the world against out-of-control AGIs?
Epimetheus30

Similarly appreciate the response!

I would say (3). Societal resilience is mandatory as threat systems proliferate and grow in power. You would need positive systems to counter them.

Regarding your points on writing in dystopia tone, I don't disagree. But it's easier to highlight an idea via narrative than bullet points. I personally like Mr. Smiles, he's my new mascot when I inevitably give up trying to solve AI alignment and turn to villainy. 

Few comparisons/contrasts on allow vs not allow creation of bad systems:

  • Major point, as above, is that disallo
... (read more)
Reply
What does it take to defend the world against out-of-control AGIs?
EpimetheusΩ24-2

I agree with Paul Christiano here. Let's call this rogue-AI preventing superintelligence Mr. Smiles. Let's assume that Mr. Smiles cannot find a "good" solution within a decade, and instead must temporarily spend much of his efforts preventing the creation of "bad" AGI.

How does Mr. Smiles ensure that no rogue actors, nation-states, corporations, or other organizations create AGI? Well, Mr. Smiles needs two things: sensors and actuators. The "mind" of Mr. Smiles isn't a huge problem, but the sensors and actuators are extremely problematic:

  1. Sensors: The world
... (read more)
Reply
3Steven Byrnes
Thanks for the comment! Right, so my concern is that humans evidently don’t take societal resilience seriously, e.g. gain-of-function research remains legal in every country on earth (as far as I know) even after COVID. So you can either: * (1) try to change that fact through conventional means (e.g. be an activist for societal resilience, either directly or via advocating for prediction markets and numeracy or something, I dunno), per Section 3.3 — I’m very strongly in favor of people working on this but don’t hold out much hope for anything more than a marginal improvement; * (2) hope that “AI helpers” will convince people to take societal resilience seriously — I’m pessimistic per the Section 3.2 argument that people won’t use AI helpers that tell them things they don’t want to hear, in situations where there are no immediate consequences, and I think sacrificing immediate gains for uncertain future societal resilience is one such area; * (3) make AIs that take societal resilience seriously and act on it, not because any human told them to but rather because their hearts are in the right place and they figured this out on their own — this is adjacent to Section 3.5.2 where we make friendly autonomous AGI, and I’m probably most optimistic / least pessimistic about that path right now; * (4) suggest that actually this whole thing is not that important, i.e., it would be nice if humans were better at societal resilience, but evidently we’ve been muddling along so far and maybe we’ll continue to do so — I’m pessimistic for various reasons in the post but I hope I’m wrong! I guess you’re suggesting (3) or (4) or maybe some combination of both, I’m not sure. You can correct me if I’m wrong. Separately, in response to your “Mr. Smiles” thing, I think all realistic options on the table can be made to sound extremely weird and dystopian. I agree with you that “AI(s) that can prevent powerful out-of-control AI from coming into existence in the first place” seems pr