Surely no competent programmer would ever advocate deploying a complex program without testing it.

With a recursively self-improving AI, once you create something able to run, running a test can turn to deploying even without programmer's intention.

Even if we manage to split the AI into modules, and test each module independently, we should understand the process enough to make sure that the individual modules can't recursively self-improve. And we should be pretty sure about the implication "if the individual modules work as we expect, then also the whole will work as we expect". Otherwise we could get a result "individual modules work OK, the whole is NOT OK and it used its skills to escape the testing environment".

My understanding is that the "appeal to authority fallacy" is specifically about appealing to irrelevant authorities. Quoting a physicist on their opinion about a physics question within their area of expertise would make an excellent non-fallacious argument. On the other hand, appealing to the opinion of say, a politician or CEO about a physics question would be a classic example of the appeal to authority fallacy. Such people's opinions would represent expert evidence in their fields of expertise, but not outside them.

I don't think the poster's description makes this clear and it really does suggest that any appeal to authority at all is a logical fallacy.

We were explicitly told to assume our knowledge is certain. Some of the circumstances assumed long chains of improbable-seeming events between sacrificing someone and saving 5 others.

Eliezer commented on this back in "Ends Don't Justify Means (Among Humans)", attempting to reconcile consequentialism with the possibility (observed in human politics) that humans may be running on hardware incapable of consequentialism accurate enough for extreme cases:

And now the philosopher comes and presents their "thought experiment"—setting up a scenario in which, by stipulation, the only possible way to save five innocent lives is to murder one innocent person, and this murder is certain to save the five lives. "There's a train heading to run over five innocent people, who you can't possibly warn to jump out of the way, but you can push one innocent person into the path of the train, which will stop the train. These are your only options; what do you do?"

An altruistic human, who has accepted certain deontological prohibits—which seem well justified by some historical statistics on the results of reasoning in certain ways on untrustworthy hardware—may experience some mental distress, on encountering this thought experiment.

So here's a reply to that philosopher's scenario, which I have yet to hear any philosopher's victim give:

"You stipulate that the only possible way to save five innocent lives is to murder one innocent person, and this murder will definitely save the five lives, and that these facts are known to me with effective certainty. But since I am running on corrupted hardware, I can't occupy the epistemic state you want me to imagine. Therefore I reply that, in a society of Artificial Intelligences worthy of personhood and lacking any inbuilt tendency to be corrupted by power, it would be right for the AI to murder the one innocent person to save five, and moreover all its peers would agree. However, I refuse to extend this reply to myself, because the epistemic state you ask me to imagine, can only exist among other kinds of people than human beings."

Now, to me this seems like a dodge. I think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. The sort of person who goes around proposing that sort of thought experiment, might well deserve that sort of answer. But any human legal system does embody some answer to the question "How many innocent people can we put in jail to get the guilty ones?", even if the number isn't written down.

As a human, I try to abide by the deontological prohibitions that humans have made to live in peace with one another. But I don't think that our deontological prohibitions are literally inherently nonconsequentially terminally right. I endorse "the end doesn't justify the means" as a principle to guide humans running on corrupted hardware, but I wouldn't endorse it as a principle for a society of AIs that make well-calibrated estimates. (If you have one AI in a society of humans, that does bring in other considerations, like whether the humans learn from your example.)

I have to admit, though, this does seem uncomfortably like the old aphorism quod licet Jovi, non licet bovi — "what is permitted to Jupiter is not permitted to a cow."

No, of course not. Against any opponent whose output has nothing to do with your previous plays (or expected plays, if they get a peak at your logic), one should clearly always defect.

I choose specks, but I found the discussion very helpful nonetheless.

Specifically, I learned that if you believe suffering is additive in any way, choosing torture is the only answer that makes sense. If you don't believe that (and I don't), then your references to "negative utility" are not as well defined as you think.

Edit: In other words, I think Torture v. Specks is just a restatement of the Repugnant Conclusion

Torture v. Specks

The problem with that one is it comes across as an attempt to define the objection out of existence - it basically demands that you assume that X negative utility spread out across a large number of people really is just as bad as X negative utility concentrated on one person. "Shut up and multiply" only works if you assume that the numbers can be multiplied in that way.

That's also the only way an interesting discussion can be held about it - if that premise is granted, all you have to do is make the number of specks higher and higher until the numbers balance out.

(And it's in no way equivalent to the trolley problem because the trolley problem is comparing deaths with deaths)

I realize I'm probably going to lose some points with you by stating this. But assuming the limit of perfect technology and the absolute correctness of the pattern theory of identity - if you can't accept these hypotheses, please just say so, instead of answering based on a different hypothesis - is there any definitive rejection of my admittedly naive notion that if you can literally read out every single atomic position, then "Chop off the head with a guillotine and drop it into a bucket of liquid nitrogen" should, yes, just work? I admit that my actual belief and assumption is that current cryonics efforts are massive overkill by people who don't realize that liquid nitrogen is not a secure encryption method for brains.

I think depicting ancient philosophers seated on a throne in heaven and the large caption "thou shalt not" sends a... somewhat mixed message about appeal to authority.

I think you misunderstood me: by "point in the story she had gotten to" I meant literally the point in the actual story (MoR). It wasn't some kind of figure of speech about her experience. (I wonder how many other people misunderstood my comment in this way; it's an interpretation that never occurred to me. I thought people knew she was a MoR reader.)

However, her experience itself was no picnic, fluff pieces notwithstanding.

I find worrying excessively over plans is often a result of not trusting the future self. People try to plan for every contingency as if as soon as the plan is finalized they will robotically follow it without reassessment. Your future self will have more data, more resources, and will have had more time to analyze. Ve will be at least as competent as you are now and probably more so. Worries about being locked into some course of action are usually overblown, and indicate that you expect to fall victim to the sunk cost fallacy in advance.

ANGTFT: future you is awesome, present you can relax a little.