Hm, thoughts are tricky things, and identity conditions of thoughts are trickier yet. I was just trying to see if you had a better idea of what "mean" might mean than me. But it seems we have to get by with what little we have.

Because I share your intuition that there is something fishy about the referential intention in Eliezer's picture. With terms like water, it's plausible that people intend to refer to "this stuff here" or "this stuff that [complicated description of their experiences with water]". With morality, it seems dubious that they should be intending to refer to "this thing that humans would all want if we were absolutely coherent etc."