I dunno if the universe can read, jmmcd. ;P

So how can the universe "enjoy itself" as much as possible before the big crunch (or before and during the heat death)*.

Maybe read the Fun Theory sequence?

It might useful to look at Pareto dominance and related ideas, and the way they are used to define concrete algorithms for multi-objective optimisation, eg NSGA2 which is probably the most used.

But that's so generic that it misses the point. Saving the world is extraordinary. What things are you doing towards extraordinary outcomes?

Any AI that would do this is unFriendly. The vast majority of uFAIs have goals incompatible with human life but not in any way concerned with it. [...] Therefore there is little to fear in the way of being tortured by an AI.

That makes no sense. The uFAIs most likely to be created are not drawn uniformly from the space of possible uFAIs. You need to argue that none of the uFAIs which are likely to be created will be interested in humans, not that few of all possible uFAIs will.

Off-topic:

I'm not talking about a basic vocabulary, but a vocabulary beyond that of the average, white, English-as-a-first-language adult.

Why white?

He ducked the question, I think, in simply saying that non-marriage was superior and/or in heaven no one is married maybe: Luke 20:27-38:

Some of the Sadducees, who say there is no resurrection, came to Jesus with a question. "Teacher," they said, "Moses wrote for us that if a man's brother dies and leaves a wife but no children, the man must marry the widow and have children for his brother. Now there were seven brothers. The first one married a woman and died childless. The second and then the third married her, and in the same way the seven died, leaving no children. Finally, the woman died too. Now then, at the resurrection whose wife will she be, since the seven were married to her?"

Jesus replied, "The people of this age marry and are given in marriage. But those who are considered worthy of taking part in that age and in the resurrection from the dead will neither marry nor be given in marriage, and they can no longer die; for they are like the angels. They are God's children, since they are children of the resurrection."

it's unclear to me how the category of "evolutionary restrictions" could apply to rationality techniques. Suggestions?

Not sure if this simple example is what you had in mind, but -- evolution wasn't capable of making us grow nice smooth erasable surfaces on our bodies, together with ink-secreting glands in our index fingers, so we couldn't evolve the excellent rationality technique of writing things down to remember them. So when writing was invented, the inventor was entitled to say "my invention passes the EOC because of the "evolutionary restrictions" clause".

Humans seem pretty good at making correct predictions even if they have made incorrect predictions in the past. More generally, any agent for whom a single wrong prediction throws everything into disarray will probably not continue to function for very long.

That's basically my point. A human has to predict the answer to questions of the type "what would I do in situation X", and their overall behavior is the sum of their actions over all situations, so they can still get the overall result roughly correct as long as they are correct on average. An AI that's capable of self-modification also has to predict the answer to questions of the type "how would my behavior be affected if I modified my decision-making algorithm in this way", where the answer doesn't just influence the behavior in one situation but all the ones that follow. The effects of individual decisions become global rather than local. It needs to be able to make much more reliable predictions if it wants to have a chance of even remaining basically operational over the long term.

Fair enough. This is an admirable habit that is all too rare, so have an upvote :).

Thanks. :)