I think that "this case will never occur in real life, thus we should ignore it when evaluating the validity of a moral theory" is a valid objection to the premises of a thought experiment, for the same reason why "this classification algorithm systematically misclassifies images of dogs as cats, but we were only going to use it for spam filtering not for image recognition so that's irrelevant" is a valid argument.