They're just conflating two different definitions of good <-- just read the part where I define Good[1] and Good[2] - the rest is specific to the comment i was replying to.

1) As they get evidence, rational agents will converge on what Good[1] is.

2) Everyone agrees that people should be Good[2]

3) Good[2] = Good[1] ...(this is the false step.)

4)Therefore, all rational agents will all want to be Good[1]

Your last post, concerning the confusion over universally compelling arguments, is similar. Just replace "good" with "mind". (As in, you are using Mind[2]="agent" while others use Mind[1]="pseudo-bounded-rational agent". Most people who use Mind[1] would class many of the things that fall in the space of Mind[2] to be objects, not agents.)

There are three Camps camps being discussed here:

People who use Mind[2] and Good[1] ...that's you and Eliezer

People who use Mind[1] and Good[2]...These are the people you are trying to understand..

People who use Mind[1] and conflate Good[1] and Good[2] ... These are the apologists who think that a sufficiently intelligent mind must behave morally. They are the only ones who are actually wrong here. Everyone else is just suffering from miscommunication because they all mean different things when they say "Mind" and "Good".