Really what where doing is looking at features such as hip ratio, symmetry, eye color and various attributes like that and that's what creates the sense of attractiveness. These are all things that the alien could measure and combine also.
Yes, this would define "looks attractive to a certain subset of humans" (i.e. those who find this set of features attractive). However, there is no such thing as "looks attractive to all humans and aliens", which is what Woman.sexiness is supposed to represent.
This comment was sitting at -2 when I saw it, which makes me think that maybe I don't understand Eliezer's point. I thought the OP was making the point that when we talk about something being "attractive" or "moral" or maybe even "sapient", we project facts about our minds into the real world. "Attractive" really means "attractive to humans", and if we forget this fact, we can end up inadvertently drawing wrong conclusions about the world. If that's wrong, then what was this post actually about?
Would a non-humanoid alien, with a different evolutionary history and evolutionary psychology, sexually desire a human female? It seems rather unlikely. To put it mildly.
People don't make mistakes like that by deliberately reasoning: "All possible minds are likely to be wired pretty much the same way, therefore a bug-eyed monster will find human females attractive." Probably the artist did not even think to ask whether an alien perceives human females as attractive. Instead, a human female in a torn dress is sexy—inherently so, as an intrinsic property.
They who went astray did not think about the alien's evolutionary history; they focused on the woman's torn dress. If the dress were not torn, the woman would be less sexy; the alien monster doesn't enter into it.
Apparently we instinctively represent Sexiness as a direct attribute of the Woman object, Woman.sexiness, like Woman.height or Woman.weight.
If your brain uses that data structure, or something metaphorically similar to it, then from the inside it feels like sexiness is an inherent property of the woman, not a property of the alien looking at the woman. Since the woman is attractive, the alien monster will be attracted to her—isn't that logical?
E. T. Jaynes used the term Mind Projection Fallacy to denote the error of projecting your own mind's properties into the external world. Jaynes, as a late grand master of the Bayesian Conspiracy, was most concerned with the mistreatment of probabilities as inherent properties of objects, rather than states of partial knowledge in some particular mind. More about this shortly.
But the Mind Projection Fallacy generalizes as an error. It is in the argument over the real meaning of the word sound, and in the magazine cover of the monster carrying off a woman in the torn dress, and Kant's declaration that space by its very nature is flat, and Hume's definition of a priori ideas as those "discoverable by the mere operation of thought, without dependence on what is anywhere existent in the universe"...
(Incidentally, I once read an SF story about a human male who entered into a sexual relationship with a sentient alien plant of appropriately squishy fronds; discovered that it was an androecious (male) plant; agonized about this for a bit; and finally decided that it didn't really matter at that point. And in Foglio and Pollotta's Illegal Aliens, the humans land on a planet inhabited by sentient insects, and see a movie advertisement showing a human carrying off a bug in a delicate chiffon dress. Just thought I'd mention that.)