That's because we have a physical truth oracle that we can do two use to test the validity of physical truths. If we could objectively observe the morality of an action, then we could have a system of morality as accurate to the perfect one as our system of science is accurate.

Right now I would figure that our system of morality (being roughly as developed as Aristotle's) bears as much relation to the proposed right system as Aristotle's science bears to the actual laws of physics. For the same reasons.

Further, I don't think that according to my current morality, I should switch my beliefs to what is right, even if I could figure out what that is. I'd have to know how to test acts for morality first, but if that test goes against my current judgement I am likely to judge the test flawed, in the same way that I would call a calculator wrong if it consistently told me that 2+2=5