"good" is determined by a currently opaque but nonetheless finite computation

With a finite definition, but not at all finite or even knowable consequences (they are knowably good, but what they are exactly, one can't know).

there's the additional claim that humanity has enough in common that this answer shouldn't vary between people any significant amount

It's going to vary a very significant amount, just a lot less than the distance from any other preference we might happen to construct, and as such, for example, creating a FAI modeled on any single person is hugely preferable for other people to letting an arbitrary AGI to develop, even if this AGI was extensively debugged and trained, and looks to possess all the right qualities.