Especially given that exposure to different fact patterns could push you in different directions. E.g. suppose right now I try to do what is right_1 (subscripts on everything to avoid appearance of claim to universality). Now, suppose that if I experience fact pattern facts_1 I conclude that it is right_1 to modify my 'moral theory' to right_2. but if I experience fact pattern facts_2 I conclude that it is right_1 to modify to right_3.

Now, that's all well and good. Eliezer would have no problem with that, as long as the diagram commutes: that is, if it's true that ( if I've experienced facts_1 and moved to right_2, and then I experience facts_2, I will move to right_4), it must also be true that ( if I've experienced facts_2 and moved to right_3, and then experience facts_1, I will move to right_4).

I suppose that at least in some cases this is true, but I see no reason why in all cases it ought to be. Especially if you allow human cognitive biases to influence the proceedings; but even if you don't (and I'm not sure how you avoid it), I don't see any argument why all such diagrams should commute. (this doesn't mean they don't, of course. I invite Eliezer to provide such an argument).

I still hold that Eliezer's account of morality is correct, except his claim that all humans would reflectively arrive at the same morality. I think foundations and priors are different enough that, functionally, each person has his own morality.