So, if I've understood you correctly, you say that the proper unit to talk about -- the thing we wish to maximize, and the thing we wish to avoid risking the loss of -- is the total number of QALYs being experienced, without reference to how many individuals are experiencing it or who those individuals are. Yes?

All right. There are serious problems with this, but as far as I can tell there are serious problems with every choice of unit, and getting into that will derail us, so I'm willing to accept your choice of unit for now in the interests of progress.

As a seperate point, I think that there isn't a consensus on what ought to be maximised is relevant.

Suppose the human species were to spread out onto 1,000,000 planets, and last for 1,000,000 years. What happens to just one planet of humans for one year is very small compared to that. Which means that anything that has even a 1% chance of making a 1% difference in the species-lifespan happiness experienced by our species is still 100,000,000 times more important than a year long delay for our one planet. It would still be 100 times more important that a year off the lifespan of the entire species.

Suppose I were the one who held the ring and, feeling the pressure of 200 lives being lost every minute, I told the AI to do whatever it thought best, or to do whatever maximised the QALYs for humanity and, thereby, set the AIs core values and purpose. An AI being benevolently inclined towards humanity, even a maginally housetrained one that knows we frown upon things like mass murder (despite that being in a good cause), is not the same as a "safe" AI or one with perfect knowledge of humanity. It might develop better knowledge of humanity later, as it grows in power, but we're talking about a fledgling just created AI that's about to have its core purpose expounded to it.

If there's any chance that the holder of the ring is going to give the AI a sub-optimal purpose (maximise the wrong thing) or leave off sensible precautions, that going the 'small step cautious milestone' approach might catch, then that's worth the delay.

But, more to the point, do we know there is a single optimal purpose for the AI to have? A single right or wrong thing to maximise? A single destiny for all species? A genetic (or computer code) template that all species will bioengineer themselves to, with no cultural differences? If there is room for a value to diversity, then perhaps there are multiple valid routes humanity might choose (some, perhaps, involving more sacrifice on humanity's part, in exchange for preserving greater divergance from some single super-happy-fun-fun template, such as valuing freedom of choice). The AI could map our options, advise on which to take for various purposes, even predict which humanity would choose, but it can't both make the choice for us, and have that option be the option that we chose for ourselves.

And if humanity does choose to take a path that places value upon freedom of choice, and if there is a small chance that how The Big Decision was made might have even a small impact upon the millions of planets and millions of years, that's a very big consequence for not taking a few weeks to move slowly and carefully.