I assume that at least some of them rely on hidden assumptions I don't see and only look like an error to me. ... I'm operating on the assumption of an inferential gap.
I don't think there is an inferential gap of the usual type (i.e. implicit hidden knowledge of facts or arguments). It's more probably a value disagreement, made harder by your objection to well-definedness of "value".
Explaining akrasia (which I do have) in terms of being mistaken what I like and having a (often unconscious) conflict between different parts of the brain works just fine for me. The moment I realize I'm not actually enjoying what I do, I either stop immediately or find that I'm fulfilling some other emotional demand, typically avoiding guilt or embarrassment.
Agreed about the unconscious conflict, but not about the conclusion. A real akrasic wants to do two incompatible things X and Y, chooses X and later regrets the choice. He knows that he will regret the choice in advance, is full aware of the problem, and still chooses X. An akrasic "enjoys" X (at the moment), but is genuinely unhappy about it later - and if he realises the problem, the unhappiness emerges already during X so that X is no longer enjoyable, but still it is hard to switch to Y. It is a real and serious problem.
The cynical (no moral judgement intended) explanation of akrasia basically tells that the agent really "prefers" X over Y, but for some reason (which usually involves hypocrisy) is mistaken about his preference. But, if it is true, tell me: why do akrasics try to fight akrasia, often privately? Why they insist that they want Y, not X, even if there are no negative consequences for admitting the desire for X? Why they are happy after doing Y and unhappy after doing X, and often remember being more happy doing Y than doing X?
Of course, you can redefine the words "want" and "prefer" to mean "what you actually do", for the price of people being mistaken about significant part of what they want. But then, these words become useless, and we lose words denoting the stuff people people report to "want" (in the conventional meaning).
(Or look at the reactions to Eliezer's Three Worlds Collide and various failed utopias.)
Failed utopias regularly fail mainly because people can't envisage all consequences of a drastic change of social order, which is caused by complexity of human societies. Being mistaken about what we want is a part of it, but not the most important one. Early communists weren't surprised that they didn't like party purges and mass executions that much. They were surprised that these things happened.
Different reactions to some fictional scenarios may well represent different preferences. Why is this explanation worse than that people are mistaken about their preferences or confused by an intuition pump? (I agree that pumps don't make reliable arguments, but sometimes they are the only accessible arguments. If you ever decide to convince me to support wireheading, you would probably need a powerful intuition pump to do it.)
You make a good point about private akrasia conflicts. I'll have to think more about this. It doesn't make sense either way right now.
The reason I object to major preference differences among humans is that this breaks with the psychological unity of humanity. It's not just that there are some minor variations or memetic hijackings in the utility function, but it seems like some are maximizing rewards, while others maximize expected utility. That's a really big difference, so it makes more sense to find an explanation that assumes only one mechanism and ex...
I've been thinking about wireheading and the nature of my values. Many people here have defended the importance of external referents or complex desires. My problem is, I can't understand these claims at all.
To clarify, I mean wireheading in the strict "collapsing into orgasmium" sense. A successful implementation would identify all the reward circuitry and directly stimulate it, or do something equivalent. It would essentially be a vastly improved heroin. A good argument for either keeping complex values (e.g. by requiring at least a personal matrix) or external referents (e.g. by showing that a simulation can never suffice) would work for me.
Also, I use "reward" as short-hand for any enjoyable feeling, as "pleasure" tends to be used for a specific one of them, among bliss, excitement and so on, and "it's not about feeling X, but X and Y" is still wireheading after all.
I tried collecting all related arguments I could find. (Roughly sorted from weak to very weak, as I understand them, plus link to example instances. I also searched any literature/other sites I could think of, but didn't find other (not blatantly incoherent) arguments.)
(There have also been technical arguments against specific implementations of wireheading. I'm not concerned with those, as long as they don't show impossibility.)
Overall, none of this sounds remotely plausible to me. Most of it is outright question-begging or relies on intuition pumps that don't even work for me.
It confuses me that others might be convinced by arguments of this sort, so it seems likely that I have a fundamental misunderstanding or there are implicit assumptions I don't see. I fear that I have a large inferential gap here, so please be explicit and assume I'm a Martian. I genuinely feel like Gamma in A Much Better Life.
To me, all this talk about "valueing something" sounds like someone talking about "feeling the presence of the Holy Ghost". I don't mean this in a derogatory way, but the pattern "sense something funny, therefore some very specific and otherwise unsupported claim" matches. How do you know it's not just, you know, indigestion?
What is this "valuing"? How do you know that something is a "value", terminal or not? How do you know what it's about? How would you know if you were mistaken? What about unconscious hypocrisy or confabulation? Where do these "values" come from (i.e. what process creates them)? Overall, it sounds to me like people are confusing their feelings about (predicted) states of the world with caring about states directly.
To me, it seems like it's all about anticipating and achieving rewards (and avoiding punishments, but for the sake of the wireheading argument, it's equivalent). I make predicitions about what actions will trigger rewards (or instrumentally help me pursue those actions) and then engage in them. If my prediction was wrong, I drop the activity and try something else. If I "wanted" something, but getting it didn't trigger a rewarding feeling, I wouldn't take that as evidence that I "value" the activity for its own sake. I'd assume I suck at predicting or was ripped off.
Can someone give a reason why wireheading would be bad?