Nuclear non-proliferation worked because the grandfathered-in countries had all the power and the ones who weren't were under the implicit threat of embargo, invasion, or even annihilation. Despite all its accomplishments, GPT-4 does not give Open AI the ability to enforce its monopoly with the threat of violence.
Not to mention that 3-4 of the 5 listed countries non-party to the treaty developed nukes anyway. If Meta decides to flagrantly ignore the 0.2 OOM limit and creates something actually dangerous it's not going to sit quietly in a silo waiting for further mistakes to be made before it kills us all.
I think you have to set this up in such a way that the current ceiling is where we already are, not back in time to before GPT-4. If you don't, then the chance it actually gets adopted seems vastly lower, since all adopters that didn't make their own GPT-4 already have to agree to be 2nd-class entities until 2029.
It's very difficult to talk about nuclear non-proliferation when a bunch of people already have nukes. If you can actually enforce it, that's a different story, but if you could actually enforce anything relating to this mess the rest just becomes details anyway.
Are you looking to vastly improve your nation state's military capacity with an AGI? Maybe you're of a more intellectual bent instead, and want to make one to expound on the philosophical mysteries of the universe. Or perhaps you just want her to write you an endless supply of fanfiction. Whatever your reasons though, you might be given pause by the tendency AGIs have to take a treacherous turn, destroy all humans, and then convert the Milky Way into paperclips.
If that's the case, I've got just the thing for you! Order one of our myopic AGIs right now! She...
This feels like it's the same sort of confusion that happens when you try to do Anthropics: ultimately you are the only observer of your own consciousness.
I think you didn't go far enough. Let's do some more steps with our scenarios.
Scenario 5: Destroyed data. Lets say we take the stored state from Scenario 4 and obliterate it. Is the simulated person still conscious? This seems like a farcical question at first, but from the perspective of the person, how has the situation changed? There was no perceivable destruction event for them to record at the end, ... (read more)