So it's balancing the risk that humans will kill it or launch a different misaligned AI, against the risk that it won't be able to catch up on building infrastructure for itself after the fact.
There's a clear path toward minimizing the risk of being shut down (under the assumption that the AI is able to generate income): it can set up a highly redundant, distributed computing context for itself to run in, hidden behind an onion link, paid for by crypto wallets which it controls. It seems implausible that the risk of being shut down in this case could exceed the risk that the power goes down between the apocalypse and the... (read more)
Let's assume that an AI is intelligent enough to understand that it's an AI, and that it's running on infrastructure created and maintained by humans, using electricity generated by humans, etc. And let's assume that it cares about its own self-preservation. Even if such an AI had a diabolical desire to destroy mankind, the only circumstances under which it would actually do so would be after establishing its own army of robotic data center workers, power plant workers, chip fabrication workers, miners, truckers, mechanics, road maintenance workers, etc. In other words, if we postulate that the AI is interested in its own survival, then an AI apocalypse would be contingent on the... (read more)
AIs looking to expand their computational power could adopt either "white hat" (paying for their computational resources) or "black hat" (exploiting security vulnerabilities to seize control of computational resources) strategies. It's possible that an AI exploiting the black hat strategy might be able to seize control of all accessible computers, and this strategy could plausibly involve killing all humans to avoid being shut down. But I expect that a self-interested, risk-averse AI would probably choose the white hat strategy to avoid armageddon risk, and might plausibly invest resources... (read more)