This is an excellent description of the argument.
Here is my question: Why bother with the middle man? No one can actually define good and everyone is constantly checking with 'human values' to see what it says! Assuming the universe runs on math and humans share attitudes about some things obviously there is some platonic entity which precisely describes human values (assuming there isn't too much contradiction) and can be called "good". But it doesn't seem especially parsimonious to reify that concept. Why add it to our ontology?
It's just semantics in a sense: but there is a reason we don't multiply entities unnecessarily.
Well, if you valued cake you'd want a way to talk about cake and efficiently distinguish cakes from non-cakes—-and especially with regards to planning, to distinguish plans that lead to cake from plans that do not. When you talk about cake there isn't really any reification of "the platonic form of cake" going on; "cake" is just a convenient word for a certain kind of confection.
The motivation for humans having a word for goodness is the same.
There seems to be a widespread impression that the metaethics sequence was not very successful as an explanation of Eliezer Yudkowsky's views. It even says so on the wiki. And frankly, I'm puzzled by this... hence the "apparently" in this post's title. When I read the metaethics sequence, it seemed to make perfect sense to me. I can think of a couple things that may have made me different from the average OB/LW reader in this regard: