is there any aspect of human existence as complicated as romance

Yes. Parenting and politics. Given a good enough model of humanity, you could probably prove that romance comes in precisely third after those two. Unlike romance, it's not even all that sensible to consider those two with non-sentient NPCs, a sign of their inherent complexity. Otherwise, good argument.

I'm coming in late, but I will say that you should probably examine the game-design literature. They are (for good commercial and aesthetic reasons) pretty much in line with your theory of fun, and in some ways advanced of it.

The problem, as I see it, is that you can't take bits out of a running piece of software and replace them with other bits, and have them still work, unless said piece of software is trivial.

The capacity to do in-place updates of running software components dates back to at least the first LISP systems. Call it 1955? Modern day telephone switches and network routers are all built with the capability of doing hot upgrades, or they wouldn't be able to reach the the level of uptime required (if you require 99.9999% uptime, going down for 30 seconds for an upgrade ruins your numbers for ten years). Additionally, those systems require that every component be independently crashable and restartable, for reliability purposes.

The best solution I've seen to the "nuke in New York" situation is that the torturers should be tried, convicted, and pardoned. The pardon is there specifically for situations where rule-based law violates perceptions of justice, but acknowledges that rule-based law and ethics should be followed first. The codification of the rule of pardon seems to conflict with the ideas of "never compromise your ethics, not even in the face of armageddon" that you are apparently advancing. Thoughts?

Because I would give odds around as extreme as the odds I would give of anything, that if you tell me "the AI you built is trying to deceive yourself", it indicates that some kind of really epic error has occurred. Controlled shutdown, immediately.

Um, no. Controlled shutdown means you are relying on software, which should be presumed corrupted, unless you are very sure about your correctness proofs. What you want there is uncontrolled shutdown, whether by pulling the plug, taking an axe to the CPU, shutting down the local power-grid, or nuking the city, as necessary. Otherwise, Hard Rapture.

>I'm looking for Dark Side epistemology itself - the Generic Defenses of Fail.

Relax. It will be over soon.

We're past that now.

X is supernatural.

X is natural.

You're correct, but it will make people uncomfortable.

You're smart. You should go to college.

>Everyone has a right to their own opinion. When you think about it, where was that proverb generated?

In the words of the great sage Emo Phillips, "I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this."

In my darker moments, I think that every human political tendency is just an instance of this very problem. Terry Pratchett (our most underrated explicator and critic of both Traditional Rationality and Bourgeois Morality) described it most pithily as "Wouldn't It Be Nice, If Everyone Was Nice". It's most obvious on the left, but can be seen on the right and libertarian tendencies as well.