What exactly is the point in deciding that 'right' is what humans brains are designed to approximate, and then being amazed that human brains approximate 'rightness'?

It's no good saying that rightness is an algorithm, and human opinions about rightness merely a crude guide. We don't have any way to refer to or define this algorithm besides looking at humans; any models we generate will always be human-behavior-derived.

Besides, talking about the algorithm is wrong. There are infinitely many algorithms that will be compatible with a finite predetermined output, and there are no restrictions on what any of them do beyond the supplied data.