...then the way that a human mind processes such arguments can be viewed as an algorithm...

What is right isn't defined by the algorithm alone but also by the environment, nurture, education and a lot of other variables. Change the circumstances and you change what is right.

You might argue that there are arguments that do not satisfy our evolutionary mind template. But that's not true, because we already encounter rules that don't feel intuitively right but that we nonetheless adopt. In such situations we call ourselves biased and want to change our minds. That's why an FAI trying to extrapolate human volition might fail for two reasons: 1.) our volition is broad and fuzzy, to an extent that right is too vague and volatile 2.) by changing the circumstances it will change what is right and therefore eventually define what is right.