Even if you only care about your map, you still care about your map as a part of the territory,

...a "territory" that exists only in your brain, since you cannot directly perceive or operate upon the real territory.

otherwise you make the next step and declare that you don't care about state of your brain either, you only care about caring itself, at which point you disappear in a "puff!" of metaphysical confusion.

...and note too, that all of this modeling is taking place in a brain. When you point to something and label it "reality", you are pointing to a portion of your map. It is not, and cannot be the actual territory. It doesn't matter how many times you try to say, "but no, I mean real reality", because the only thing you have to represent that idea with is your brain.

It is still the idea of reality, a model of reality, that exists in your map.

Yes, the map is represented physically within the territory. The map "runs on" a platform made of territory, just as "you" run on a platform made of map. All I am saying is, "you" cannot access the territory directly, because your platform is made of map, just as the map's platform is made of territory. Everything we perceive or think or imagine is therefore map... including the portion of the map we refer to as the territory.

We want our preferences to point to the territory, but they cannot do so in actual reality. We have the illusion that they can, because we have the same representative freedom as an artist drawing a picture of hands drawing each other - we can easily represent paradoxical and unreal things within the surface of the map.

(I remember reading at one point a tutorial in General Semantics that illustrated this point much better than I am doing, but sadly I cannot seem to find it at the moment.)

The comments about people's differences in not-being-deceived were just making the point that that preference is more about consensus reality than reality reaity [sic].

I'm afraid I have no idea what this is supposed to mean. It seems to me like you're just repeating your conclusion over and over again using different words, which unfortunately doesn't constitute an argument. Maybe to you it seems like we're doing the same thing, I don't know.

Alternatively, maybe we're still talking past each other for the reasons suggested here (which everyone seemed to agree with at the time.) In which case, I wonder why we're still having this conversation at all, and apologise for my part in pointlessly extending it. ;)

You could still write a meta-reply, taking that problem into account. The root of the disagreement can be stated in one line, and a succinct statement of at the moment unresolvable disagreement is resolution of an argument.

Wow, -5! People here don't seem to appreciate this sort of challenge to their conceptual framework.

I'm just saying that human beings have no way to model their preferences except by modeling experiences, in human-sensory terms.

I agree, but I wonder if I failed to communicate the distinction I was attempting to make. The human-sensory experience of being embedded in a concrete, indifferent reality is (drugs, fantasies, and dreams aside) basically constant. It's a fundamental thread underlying our entire history of experience.

It's this indifference to our mental state that makes it special. A preference expressed in terms of "reality" has subjective properties that it would otherwise lack. Maybe I want the sky to be blue so that other people will possess a similar experience of it that we can share. "Blueness" may still be a red herring, but my preference now demands some kind of invariant between minds that seemingly cannot be mediated except through a shared external reality. You might argue that I really just prefer shared experiences, but this ignores the implied consistency between such experiences and all other experiences involving the external reality, something I claim to value above and beyond any particular experience.

Even if you phrase this as, "I prefer the sky to be actually blue, even if I don't know it", it is still a lie, because now you are modeling an experience of the sky being blue, plus an experience of you not knowing it.

This is where the massive implicit context enters the scene. "Even if I don't know it" is modeled after experience only in the degenerate sense that it's modeled after experience of indifferent causality. A translation might look like "I prefer to experience a reality with the sorts of consequences I would predict from the sky being blue, even if I don't consciously perceive blue skies". That's still an oversimplification, but it's definitely more complex than just invoking a generic memory of "not having known something" and applying it to blue skies.

The two notions are basically isomorphic, so where's the value in the distinction?

Well, it makes clear some of the limits of certain endeavors that are often discussed here. It dissolves confusions about the best ways to make people happy, and whether a world should be considered "real" or "virtual", and whether it's somehow "bad" to be virtual.

I don't see how any of that is true. I can easily think of different concrete realizations of "real" and "virtual" that would interact differently with my experience of reality, thus provoking different labellings of "good" and "bad". If your point is merely that "real" is technically underspecified, then I agree. But I don't see how you can draw inferences from this underspecification.

For another person, it might be associated with the blinding heat of the desert and a sensation of thirst... and these two people can then end up arguing endlessly about whether a blue sky is obviously good or bad.

And both are utterly deluded to think this their preferences have anything to do with reality.

I'm going to have to turn your own argument against you here. To the extent that you have a concept of reality that is remotely consistent with your everyday experience, I claim that "in reality, blue skies are bad because they provoke suffering" is a preference stated in terms of an extremely similar reality-concept, plus a suffering-concept blended together from first-hand experience and compassion (itself also formed in terms of reality-as-connected-to-other-minds). For you to say it has "nothing to do with reality" is pure semantic hogwash. What definition of "reality" can you possibly be using to make this statement, except the one formed by your lifetime's-worth of experience with indifferent causality? You seem to be denying the use of the term to relate your concept of reality to mine, despite their apparent similarity.

However, to the extent that our preferences produce negative experiences, it is saner to remove the negative portion of the preference.

This doesn't make sense to me. Whether or not an experience is "negative" is a function of our preferences. If a preference "produces" negative experiences, then either they're still better than the alternative (in which case it's a reasonable preference, and it's probably worthwhile to change your perception of the experience) or they're not (in which case it's not a true preference, just delusion).

Luckily, human beings are not limited to, or required to have, bidirectional preferences. Feeling pain at the absence of something is not required in order to experience pleasure at its presence, in other words. (Or vice versa.)

That's a property of pain and pleasure, not preference. I may well decide not to feel pain due to preference X being thwarted, but I still prefer X, and I still prefer pleasure to the absence of pleasure.

Awareness of this fact, combined with an awareness that it is really the experience we prefer (and mainly, the somatic markers we have attached to the experience) makes it plain that the logical thing to do is to remove the negative label, and leave any positive labels in place.

This is where I think your oversimplification of "experience vs reality" produces invalid conclusions. Those labels don't just apply to one experience or another, they apply to a massively complicated network of experience that I can't even begin to hold in my mind at once. Given that, your logic doesn't follow at all, because I really don't know what I'm relabeling.

This relates to a general reservation I have with cavalier attitudes toward mind-hacks: I know full well that my preferences are complex, difficult to understand, and grossly underspecified in any conscious realization, so it's not at all obvious to me that optimizing a simple preference concerning one particular scenario doesn't carry loads of unintended consequences for the rest of them. I've had direct experience with my subconsciously directed behavior "making decisions for me" that I had conscious reasons to optimize against, only later to find out that my conscious understanding of the situation was flawed and incomplete. I think that ignoring the intuitive implications of an external reality leads to similar contradictions.

Agreed, as the brain is a physical object. Referring "directly" to subjective experiences is a different story though.

Presumably, if you asked such an agent to reflect on its own purposes, it would claim that they related to the external world (unless it's aim was to deceive you about its purposes for signalling reasons, of course).

For example, it might claim that its aim was to save the whales - rather than to feel good about saving the whales. It could do the latter by taking drugs or via hypnotherapy - and that is not how it actually acts.

I'm hard pressed, though, to conceive of a moral philosophy where improved health would not be considered "good for you".

Preference utilitarianism applied to someone who thinks that it is only through suffering that life can achieve meaning.

To be clear, I don't subscribe to such a view myself, but it's conceivable. I agree with you that health is good for people. My point is just that this agreement owes more to shared intuition than conclusive empirical testing.