We were explicitly told to assume our knowledge is certain. Some of the circumstances assumed long chains of improbable-seeming events between sacrificing someone and saving 5 others.
Eliezer commented on this back in "Ends Don't Justify Means (Among Humans)", attempting to reconcile consequentialism with the possibility (observed in human politics) that humans may be running on hardware incapable of consequentialism accurate enough for extreme cases:
And now the philosopher comes and presents their "thought experiment"—setting up a scenario in which, by stipulation, the only possible way to save five innocent lives is to murder one innocent person, and this murder is certain to save the five lives. "There's a train heading to run over five innocent people, who you can't possibly warn to jump out of the way, but you can push one innocent person into the path of the train, which will stop the train. These are your only options; what do you do?"
An altruistic human, who has accepted certain deontological prohibits—which seem well justified by some historical statistics on the results of reasoning in certain ways on untrustworthy hardware—may experience some mental distress, on encountering this thought experiment.
So here's a reply to that philosopher's scenario, which I have yet to hear any philosopher's victim give:
"You stipulate that the only possible way to save five innocent lives is to murder one innocent person, and this murder will definitely save the five lives, and that these facts are known to me with effective certainty. But since I am running on corrupted hardware, I can't occupy the epistemic state you want me to imagine. Therefore I reply that, in a society of Artificial Intelligences worthy of personhood and lacking any inbuilt tendency to be corrupted by power, it would be right for the AI to murder the one innocent person to save five, and moreover all its peers would agree. However, I refuse to extend this reply to myself, because the epistemic state you ask me to imagine, can only exist among other kinds of people than human beings."
Now, to me this seems like a dodge. I think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. The sort of person who goes around proposing that sort of thought experiment, might well deserve that sort of answer. But any human legal system does embody some answer to the question "How many innocent people can we put in jail to get the guilty ones?", even if the number isn't written down.
As a human, I try to abide by the deontological prohibitions that humans have made to live in peace with one another. But I don't think that our deontological prohibitions are literally inherently nonconsequentially terminally right. I endorse "the end doesn't justify the means" as a principle to guide humans running on corrupted hardware, but I wouldn't endorse it as a principle for a society of AIs that make well-calibrated estimates. (If you have one AI in a society of humans, that does bring in other considerations, like whether the humans learn from your example.)
I have to admit, though, this does seem uncomfortably like the old aphorism quod licet Jovi, non licet bovi — "what is permitted to Jupiter is not permitted to a cow."
But since I am running on corrupted hardware, I can't occupy the epistemic state you want me to imagine.
It occurs to me that many (maybe even most) hypotheticals require you to accept an unreasonable epistemic state. Even something so simple as trusting that Omega is telling the truth [and that his "fair coin" was a quantum random number generator rather than, say, a metal disc that he flipped with a deterministic amount of force, but that's easier to grant as simple sloppy wording]
It is a common part of moral reasoning to propose hypothetical scenarios. Whether it is our own Torture v. Specks or the more famous Trolley problem, asking these types of questions helps the participants formalize and understand their moral positions. Yet one common response to hypothetical scenarios is to challenge some axiom of the problem. This article is a request that people stop doing that, and an explanation of why this is an error.
First, a brief digression into law, which is frequently taught using hypothetical questioning. Under the Model Penal Code:
A person acts knowingly with respect to a material element of an offense when:
(i) if the element involves the nature of his conduct or the attendant
circumstances, he is aware that his conduct is of that nature or that such
circumstances exist; and
(ii) if the element involves a result of his conduct, he is aware that it is
practically certain that his conduct will cause such a result.
Hypothetical: If Bob sets fire to a house with Charlie inside, killing Charlie, is Bob guilty of knowing killing of Charlie? Bob genuinely believes throwing salt over one's shoulder when one sets a building on fire protects all the inhabitants and ensures that they will not be harmed - and did throw salt over his shoulder in this instance.
Let us take it as a given that setting someone on fire is practically certain to kill them. Nonetheless, Bob did not knowingly kill Charlie because Bob was not aware of the consequence of his action. Bob had a false belief that prevented him from having the belief required under the MPC to show knowledge.
The obvious response here is that, in practice, the known facts will lead to Bob's conviction of the crime at trial. This is irrelevant. Bob will be convicted at trial because the jury will not believe Bob's asserted belief was true. Unless Bob is insane or mentally deficient, the jury would be right to disbelieve Bob. But that missed the point of the hypothetical.
The purpose of the hypothetical is to distinguish between one type of mental state and a different type of mental state. If you don't understand that Bob is innocent of knowing killing if he truly believed that Charlie was safe, then you don't understand the MPC definition of knowing. Discussion of how mental states are proven at real trials, or whether knowing killing should be the only criminal statute about killing are different topics. Talking about those topics will not help you understand the MPC definition of knowing. Talking about those other topics is functionally identical to saying that you don't care about understanding the MPC definition of knowing.
Likewise, people who responds to the Trolley problem by saying that they would call the police are not talking about the moral intuitions that the Trolley problem intends to explore. There's nothing wrong with you if those problems are not interesting to you. But fighting the hypothetical by challenging the premises of the scenario is exactly the same as saying, "I don't find this topic interesting for whatever reason, and wish to talk about something I am interested in."
In short, fighting the premises of a hypothetical scenario is changing the topic to focus on something different than topic of conversation intended by the presenter of the hypothetical question. When changing the topic is appropriate is a different discussion, but it is obtuse to fail to notice that one is changing the subject.
Edit: My thesis "Notice and Justify changing the subject," not "Don't change the subject."