So you think that true moral behavior excludes choice? (More generally, once someone chooses their morality, no more choices remain to be made?)
I think so. What choice is there in the field mathematics? I don't see that mathematicians ever had any choice but to eventually converge at the same answer given the same conjecture. Why would that be different given an objective morality?
I thought that is what the Aumann's agreement theorem states and the core insight of TDT, that rational agents will eventually arrive at the same conclusions and act accordingly.
The question would be what happens if a superintelligence was equipped with goals that contradict reality. If there exists an objective morality and the goal of a certain AI was to maximize paperclips while maximizing paperclips was morally wrong then that would be similar to the goal of proving that 1+1=3 or to attain faster-than-light propagation.
ETA I suppose that if there does exist a sort of objective morality but it is inconsistent with its goals, then you would end up with an unfriendly AI anyway. Since such an AI would attempt to pursue its goals given the small probability that there is no objective morality for one reason or the other.
I think so. What choice is there in the field mathematics? I don't see that mathematicians ever had any choice but to eventually converge at the same answer given the same conjecture.
Mathematics is not an agent, it cannot be controlled anyway. But mathematicians have choice over what branch of math to pursue.
Just a minor thought connected with the orthogonality thesis: if you claim that any superintelligence will inevitably converge to some true code of morality, then you are also claiming that no measures can be taken by its creators to prevent this convergence. In other words, the superintelligence will be uncontrollable.