Perhaps, but absolute power tends to be the more relevant one, as it definitionally also includes the means to persue the goals derived from absolute corruption.

I wonder where one could apply "Absolute" and not come up with a scary sounding conclusion. Absolute skepticism seems it would turn one into a gibbering madman. Absolute logic--well what is a dangerous AI but absolute logic plus power?

IIRC the standard experimental result is that atheists who were raised religious have substantially above-average knowledge of their former religions. I am also suspicious that any recounting whatsoever of what went wrong will be greeted by, "But that's not exactly what the most sophisticated theologians say, even if it's what you remember perfectly well being taught in school!"

This obviously won't be true in my own case since Orthodox Jews who stay Orthodox will put huge amounts of cumulative effort into learning their religion's game manual over time. But by the same logic, I'm pretty sure I'm talking about a very standard element of the religion when I talk about later religious authorities being presumed to have immensely less theological knowledge than earlier authorities and hence no ability to declare earlier authorities wrong. As ever, you do not need a doctorate in invisible sky wizard to conclude that there is no invisible sky wizard, and you also don't need to know all the sophisticated excuses for why the invisible sky wizard you were told about is not exactly what the most sophisticated dupes believe they believe in (even as they go on telling children about the interventionist superparent). It'd be nice to have a standard, careful and correct explanation of why this is a valid attitude and what distinguishes it from the attitude of an adolescent who finds out everything they were told about quantum mechanics is wrong, besides the obvious distinction of net weight of experimental evidence (though really that's just enough).

LW has reportedly been key in deconverting many, many formerly religious readers. Others will of course have fled. It takes all kinds of paths.

I think it is a comment on the tendency of human minds to model complex systems as simple ones and therefore stick strongly to a few remedies whether they are sensible or not - ancestrally "whack it with a club" but in the case of computers, "reboot it", "run the virus scanner" and "defrag it". Admittedly, for old computers that relied on vacuum tubes whose connections would sometimes work loose, "whack it with a club" did, in fact, occasionally work.

Arrives late to the party

Really great story, iceman. Some comments:

*Running the story through a beta group of non-LW bronies would definitely be a good idea to catch which ideas may need more explanation.

*I really like how it's repeatedly show that when you interact with a super-intelligence, even if it's just free conversation, the state of mind you leave in is probably going to be the state of mind it wants you to leave in. As others have said, this could be driven home even stronger by showing CelestAI strongly tailoring her interaction to different humans. Maybe even have her directly contradict herself in the information she uses to persuade people to upload.

*Related to the above, I can imagine a non-LW reader getting to chapter 5 and forever after wondering why Lars or anyone else never tries to shut Celestia down. I'm not sure how obvious an intuition it is that by the time you notice a super-intelligence doing things that make you want to stop it, it's probably already far too late to stop it. In this case I assumed Celestia would have made sure that she is already invincible by the standards of human technology before launching any plan that's going to antagonize people, but an unsophisticated reader might not get that.

*This may just be me, but I'd prefer a bit more closure on the cosmic scale story. What is Celestia's plan against running out of matter? Slow the clock speed of her servers over time? Bust open the physics textbooks and hope for a useful paradigm shift?

Anyway, very good stuff.

Shouldn't it really be "Believing that FTL is impossible is undesirable iff FTL is possible"?

You seemed to be doing something clever with quotes, but mostly that made it hard to read. :P

I don't get it.

Actually, it should be "FTL is impossible" is undesirable if and only if FTL is possible."

FTL being impossible is undesirable if you want to go to the stars.

The conclusion that "FTL is impossible" is undesirable if and only iff FTL is possible.

The two conditions are very different.

As anyone who's ever moderated a large forum can tell you, playing with incentives works almost embarrassingly well and quickly compared to working on sympathy or respect for authority. Of course, it's also harder to do.

That sounds very intriguing. Can you give some example of how you've used "playing with incentives" successfully to (I assume - correct me if I'm wrong) maintain a productive forum? That might be very enlightening - seriously, no irony here.

He shook his head. "No, for the purposes of this discussion, Asuka... only I have the power to decide humanity's fate. And I refuse that power to give it back to them. Humanity is made of neither heaven or hell; that with freedom of choice and honor, as though the maker and molder of itself... that they may fashion themselves in whatever form they shall prefer. People, individuals, are not single things but always tip from order to chaos and back again. Those with order are needed for stability. Those who espouse chaos bring change. Only humanity may balance humanity. If a God should be needed, only that nothing from without should threaten that free choice. The maker should be the first servant, just as the mother would care without hesitation for her child

-Shinji & Warhammer Xover.