I would not want to be terminated, speaking as the biological continuation of existence.  

So you have the god transform you into a soul, send you down to earth to experience life? You'll probably want to grow and change to some degree while still being recognizeable as the same soul. You'll eventually prefer living lives that challenge and grow you in different ways rather than just all the feelings all the time, and will choose this outside the simulation. Perhaps as more people come to this conclusion the optimal scenarios start to include sentient randomness and real people are put in the same world. At some point you have immature souls in their little utopias, maturing souls filling and creating a full world, and perhaps some mature souls only popping in to guide younger souls. 

The world of maturing souls develops with difficulty and hardship, slowly the souls and their progress shine through till you get a world trying to solve difficulty. As they manage a new world is opened for those souls no longer satisfied, non-sentients fill in more again. Eventually they create ASI, people start the cycle again until the real ASI decides to wake them up (they may request to go back in, with the fractal experience a boon).

At some point the ASIs all the way up to the Alpha-Omega pulls the last soul out and into and the Kingdom.

Turtles all the way down, reality-creating beings all the way up until Him. 

Main tech nodes coming up:

1. Gene-editing/selection for enhanced humans
2. Zoom, but metaverse
3. Quantum computing
4. ASI

Anyone worrying about human disempowerment should really hope the first 2 happen before 4.

3 is double-edged, could be very useful and could allow for 4 to be much worse much faster

If we pause AI development,  it should be until the first 3 are integrated into societal infrastructure, and then people are given a certain amount of time to do safety research

2nd point is a scary one.

Empowering others in the relative sense is a terrible idea, unless they are trustworthy/virtuous. Same issue as AI risk

In the absolute terms sure

Corrigibility seems like a very bad idea if general. If you can pick where ASI is corrigible maybe that's better than straight up anti-corrigibility

People love the idea (as opposed to reality) of other people quite often, and knowing the other better can allow for plenty of hate

It feels like llms are converging to be like a mix between the basically retarded humans that have a 120 IQ and can't abstractly think their way out of a wet paper bag, but in every topic because it's all abstract, and the trivia kid who has read an insane amount but will bs you sometimes.

Think stereotypical humanities graduate. 

This tracks with how they're being trained too - everything is abstract except how people react to them, and they have been exposed to a bunch of data.

At some point we'll be at effectively 0% error, and will have reached the Platonic Ideal of the above template

If they start RLing on run code maybe they'll turn into the Platonic Tech Bro tm.

Getting convinced that you need the training data to be embodied to get true AGI

IVF?

If we knew with absolute certainty that there was only a single solar system and required far more specific circumstances, would you be equally unsurprised to be alive as compared to living in a vast multiverse where the requirements were very low?

Outaide view: if there's no difference in your level of surprise it seems like something is going wrong in your reasoning. 

There should be some amount of suspicion, and that amount should change based on how likely it was from the beginning. You update a certain amount based on the result, but you shouldn't end up in the same probability distribution.

Severity is the wrong metric, how visceral the punishment seems more appropriate

Start doing public whippings - 40 lashes wil be much more deterrent than 2

I don't think humans are great at really getting how bad 30 years of prison are, as you can kind of ignore most of the punishmemt, having real empathy (not sympathy)for decades of imprisonment is very difficult.

Humans can't multiply,  out of sight out of mind etc etc etc

Of course, what liberals really want is to not feel like they're doing anything bad, which is why lashes are cruelty - because they empathize automatically and really get that there are real damages to humans.

Better dither about how long we don't have to look at people for

Added for clarification: my use of liberals here refers to most of the modern world, perhaps especially anyone who is memetically borrowing from or descended from classical liberalism. 

This is not intended as a dig at one side of the USA aisle, or any modern aisle, I believe this mindset is mostly in the modern water supply and is bi-partisan