Wow, -5! People here don't seem to appreciate this sort of challenge to their conceptual framework.

I'm just saying that human beings have no way to model their preferences except by modeling experiences, in human-sensory terms.

I agree, but I wonder if I failed to communicate the distinction I was attempting to make. The human-sensory experience of being embedded in a concrete, indifferent reality is (drugs, fantasies, and dreams aside) basically constant. It's a fundamental thread underlying our entire history of experience.

It's this indifference to our mental state that makes it special. A preference expressed in terms of "reality" has subjective properties that it would otherwise lack. Maybe I want the sky to be blue so that other people will possess a similar experience of it that we can share. "Blueness" may still be a red herring, but my preference now demands some kind of invariant between minds that seemingly cannot be mediated except through a shared external reality. You might argue that I really just prefer shared experiences, but this ignores the implied consistency between such experiences and all other experiences involving the external reality, something I claim to value above and beyond any particular experience.

Even if you phrase this as, "I prefer the sky to be actually blue, even if I don't know it", it is still a lie, because now you are modeling an experience of the sky being blue, plus an experience of you not knowing it.

This is where the massive implicit context enters the scene. "Even if I don't know it" is modeled after experience only in the degenerate sense that it's modeled after experience of indifferent causality. A translation might look like "I prefer to experience a reality with the sorts of consequences I would predict from the sky being blue, even if I don't consciously perceive blue skies". That's still an oversimplification, but it's definitely more complex than just invoking a generic memory of "not having known something" and applying it to blue skies.

The two notions are basically isomorphic, so where's the value in the distinction?

Well, it makes clear some of the limits of certain endeavors that are often discussed here. It dissolves confusions about the best ways to make people happy, and whether a world should be considered "real" or "virtual", and whether it's somehow "bad" to be virtual.

I don't see how any of that is true. I can easily think of different concrete realizations of "real" and "virtual" that would interact differently with my experience of reality, thus provoking different labellings of "good" and "bad". If your point is merely that "real" is technically underspecified, then I agree. But I don't see how you can draw inferences from this underspecification.

For another person, it might be associated with the blinding heat of the desert and a sensation of thirst... and these two people can then end up arguing endlessly about whether a blue sky is obviously good or bad.

And both are utterly deluded to think this their preferences have anything to do with reality.

I'm going to have to turn your own argument against you here. To the extent that you have a concept of reality that is remotely consistent with your everyday experience, I claim that "in reality, blue skies are bad because they provoke suffering" is a preference stated in terms of an extremely similar reality-concept, plus a suffering-concept blended together from first-hand experience and compassion (itself also formed in terms of reality-as-connected-to-other-minds). For you to say it has "nothing to do with reality" is pure semantic hogwash. What definition of "reality" can you possibly be using to make this statement, except the one formed by your lifetime's-worth of experience with indifferent causality? You seem to be denying the use of the term to relate your concept of reality to mine, despite their apparent similarity.

However, to the extent that our preferences produce negative experiences, it is saner to remove the negative portion of the preference.

This doesn't make sense to me. Whether or not an experience is "negative" is a function of our preferences. If a preference "produces" negative experiences, then either they're still better than the alternative (in which case it's a reasonable preference, and it's probably worthwhile to change your perception of the experience) or they're not (in which case it's not a true preference, just delusion).

Luckily, human beings are not limited to, or required to have, bidirectional preferences. Feeling pain at the absence of something is not required in order to experience pleasure at its presence, in other words. (Or vice versa.)

That's a property of pain and pleasure, not preference. I may well decide not to feel pain due to preference X being thwarted, but I still prefer X, and I still prefer pleasure to the absence of pleasure.

Awareness of this fact, combined with an awareness that it is really the experience we prefer (and mainly, the somatic markers we have attached to the experience) makes it plain that the logical thing to do is to remove the negative label, and leave any positive labels in place.

This is where I think your oversimplification of "experience vs reality" produces invalid conclusions. Those labels don't just apply to one experience or another, they apply to a massively complicated network of experience that I can't even begin to hold in my mind at once. Given that, your logic doesn't follow at all, because I really don't know what I'm relabeling.

This relates to a general reservation I have with cavalier attitudes toward mind-hacks: I know full well that my preferences are complex, difficult to understand, and grossly underspecified in any conscious realization, so it's not at all obvious to me that optimizing a simple preference concerning one particular scenario doesn't carry loads of unintended consequences for the rest of them. I've had direct experience with my subconsciously directed behavior "making decisions for me" that I had conscious reasons to optimize against, only later to find out that my conscious understanding of the situation was flawed and incomplete. I think that ignoring the intuitive implications of an external reality leads to similar contradictions.

Agreed, as the brain is a physical object. Referring "directly" to subjective experiences is a different story though.

Presumably, if you asked such an agent to reflect on its own purposes, it would claim that they related to the external world (unless it's aim was to deceive you about its purposes for signalling reasons, of course).

For example, it might claim that its aim was to save the whales - rather than to feel good about saving the whales. It could do the latter by taking drugs or via hypnotherapy - and that is not how it actually acts.