Well, for a start, I don't think that the builders would want to be the only people in their world. And recall that this only serves to produce new humans, because simply making all existing humans immortal solves the DA as well. I think it would be more efficient to fully populate the simulation.
What is this decision theory? I haven't read the Sequences yet, sorry.
because simply making all existing humans immortal solves the DA as well.
I disagree. An appreciable number of people might be the ones designing the AI, but they won't spend an appreciable portion of their lives doing it.
A self-modifying AI is built to serve humanity. The builders know, of course, that this is much riskier than it seems, because its success would render their own observations extremely rare. To solve the problem, they direct the AI to create billions of simulated humanities in the hope that this will serve as a Schelling point to them, and make their own universe almost certainly simulated.
Plausible?