About failed utopias, you misunderstood me.
Fair enough.
it seems like some are maximizing rewards, while others maximize expected utility
Utility is so general a term that it can encompass rewards. It can be said that all people are maximising utility whenever their decisions don't exhibit cyclic preferences or some other blatant (but nevertheless common) error, but this would also be a bit misleading - recalling the von Neumann-Morgenstern theorem usually begs for the cynic interpretation of utility that does care more about what people do rather than what they really want.
It's probably better to say that there are at least two distinct decision processes or systems working together in the brain, and, depending on circumstances, one of them prevails. The unconscious process steers the decision towards safe immediate psychological rewards; the conscious one plans further in advance and tries to accomplish more complex aims related to the external world. (Generalisation to the case of more than two processes working on several different time scales should be straightforward.)
Sometimes - in stress, during akrasic behaviour, presumably also under wireheading, the unconscious system overrides the conscious one and executes its commands. In other situations the conscious system can take priority. The conscious system wants to remain in control, but knows that it can be overriden. Therefore it tries to avoid situations where that can happen.
Now into the more speculative realm. I would guess that retaining at least some control should be strongly prioritised over any amount of pleasure on the level of the conscious system, and that this may even be a human universal. But the conscious mind can be fooled into thinking that the control will not be lost in spite of a real danger. For example, the drug addicts overwhelmingly report that they can always stop - when they finally realise that it is not the case, the relevant part of their behaviour is already firmly controlled by the unconscious mind.
The rejection of wireheading may be the manifestation of the desire of the conscious mind to remain in control. Wireheading was traditionally described as total dictatorship of the unconscious mind, and is therefore rejected whenever the conscious mind is under control. But there is a way to overcome that: present wireheading in a different way, more akin to computer game worlds than to heroin. Computer games are basically safe for most users - there were cases of people dying during play, but these are rare. The conscious mind may think that the wireheading will simply be analogous to moving to a different country and that the control will not be lost. So that may be the reason for differing opinions - different intuitions about wireheading. We don't know how would wireheading feel and so it's hardly surprising that the intuitions differ.
But even if it were not true, well, some people move abroad leaving their families and friends and jobs behind, others can't imagine that. Does it break the psychological unity of humanity? There were people who didn't leave their country even if it was the only real chance to save their lives. Why do you expect that we will all agree on a hypothetical whose role in our evolution is non-existent and which belongs to the class of things which we consistently can't reason well about, when we differ in more mundane (and therefore evolutionary salient) decisions?
Now into the more speculative realm. I would guess that retaining at least some control should be strongly prioritised over any amount of pleasure on the level of the conscious system, and that this may even be a human universal.
(I'm not fully convinced of the conscious/unconscious split you outline, but let's go with it for the sake of the argument. It's certainly a reasonable hypothesis.)
Why would you side with the conscious mind? Do you have a specific reason for this, besides "because it's the one that holds the power" (which is perfectly...
I've been thinking about wireheading and the nature of my values. Many people here have defended the importance of external referents or complex desires. My problem is, I can't understand these claims at all.
To clarify, I mean wireheading in the strict "collapsing into orgasmium" sense. A successful implementation would identify all the reward circuitry and directly stimulate it, or do something equivalent. It would essentially be a vastly improved heroin. A good argument for either keeping complex values (e.g. by requiring at least a personal matrix) or external referents (e.g. by showing that a simulation can never suffice) would work for me.
Also, I use "reward" as short-hand for any enjoyable feeling, as "pleasure" tends to be used for a specific one of them, among bliss, excitement and so on, and "it's not about feeling X, but X and Y" is still wireheading after all.
I tried collecting all related arguments I could find. (Roughly sorted from weak to very weak, as I understand them, plus link to example instances. I also searched any literature/other sites I could think of, but didn't find other (not blatantly incoherent) arguments.)
(There have also been technical arguments against specific implementations of wireheading. I'm not concerned with those, as long as they don't show impossibility.)
Overall, none of this sounds remotely plausible to me. Most of it is outright question-begging or relies on intuition pumps that don't even work for me.
It confuses me that others might be convinced by arguments of this sort, so it seems likely that I have a fundamental misunderstanding or there are implicit assumptions I don't see. I fear that I have a large inferential gap here, so please be explicit and assume I'm a Martian. I genuinely feel like Gamma in A Much Better Life.
To me, all this talk about "valueing something" sounds like someone talking about "feeling the presence of the Holy Ghost". I don't mean this in a derogatory way, but the pattern "sense something funny, therefore some very specific and otherwise unsupported claim" matches. How do you know it's not just, you know, indigestion?
What is this "valuing"? How do you know that something is a "value", terminal or not? How do you know what it's about? How would you know if you were mistaken? What about unconscious hypocrisy or confabulation? Where do these "values" come from (i.e. what process creates them)? Overall, it sounds to me like people are confusing their feelings about (predicted) states of the world with caring about states directly.
To me, it seems like it's all about anticipating and achieving rewards (and avoiding punishments, but for the sake of the wireheading argument, it's equivalent). I make predicitions about what actions will trigger rewards (or instrumentally help me pursue those actions) and then engage in them. If my prediction was wrong, I drop the activity and try something else. If I "wanted" something, but getting it didn't trigger a rewarding feeling, I wouldn't take that as evidence that I "value" the activity for its own sake. I'd assume I suck at predicting or was ripped off.
Can someone give a reason why wireheading would be bad?