2: Machine does not allow interaction with other real people. (Less-trivially fixable, but still very fixable. Networked MBLSes would do the trick, and/or ones with input devices to let outsiders communicate with folks who were in them.
How could you tell the difference? Let's say I claim to have build a MBLS that doesn't contain any sentients whatsoever and invite you to test it for an hour. (I guarantee you it won't rewire any preferences or memories; no cheating here.) Do you expect to not be happy? I have taken great care that emotions like loneliness or guilt won't arise and that you will have plenty of fun. What would be missing?
Like in my response to Yasuo, I find it really weird to distinguish states that have no different experiences, that feel exactly the same.
Let's consider another case: suppose my neurochemistry were altered so I just had a really high happiness set point [...] but had comparable emotional range to what I have now [...] so I could dip low when unpleasant things happened [...]
Why would you want that? To me, that sounds like deliberately crippling a good solution. What good does it do to be in a low mood when something bad happens? I'd assume that this isn't an easy question to answer and I'm not calling you out on it, but "I want to be able to feel something bad" sounds positively deranged.
(I can see uses with regards to honest signaling, but then a constant high set-point and a better ability to lie would be preferable.)
It does not seem like a transmuted orgasmium version of "me" would remember much [...]. Remembering things is not universally enjoyable, and anyway it's rarely the most enjoyable thing I could be doing; this faculty would be replaced.
Yes, I would imagine orgasmium to essentially have no memory or only insofar as it's necessary for survival and normal operations. Why does that matter? You already have a very unreliable and sparse memory. You wouldn't lose anything great in orgasmium; it would always be present. I can only think of the intuition "the only way to access some of the good things that happened to me, right now, is through my memory, so if I lost it, those good things would be gone". Orgasmium is always amazing.
But then, that can't be exactly right, as you say you'd be more at ease to have memory you simply never use. I can't understand this. If you don't use it, how can it possibly affect your well-being, at any point? How can you value something that doesn't have a causal connection to you?
I think in general this boils down to: I don't want to lose capacities that I currently have.
How do you know that? I'm not trying to play the postmodernism card "How do we know anything?", I'm genuinely curious how you arrived at this conclusion. If I try to answer the question "Do I care about losing capacities?", I go through thought experiments and try to imagine scenarios that are only distinguished by the amount of capacities I have and then see what emotional reaction comes up. But then I'm still answering the question based on my (anticipated and real) rewards, so I'm really deciding what state I would enjoy more and pick the more enjoyable one (or less painful one). Wireheading, however, is always maximally enjoyable, so it seems I should always choose it.
(For completeness, I would normally agree with you that losing capacities is bad, but only because losing optimization power makes it harder to arrive at my goals. If I saw no need for more power, e.g. because I'm already maximally happy and there's a system to ensure sustainability, I'd happily give up everything.)
(Finally, I really appreciate your detailed and charitable answer.)
Apologies for coming to the discussion very, very late, but I just ran across this.
If I saw no need for more power, e.g. because I'm already maximally happy and there's a system to ensure sustainability, I'd happily give up everything.
How could you possibly get into this epistemic state? That is, how could you possibly be so sure of the sustainability of your maximally happy state, without any intervention from you, that you would be willing to give up all your optimization power?
(This isn't the only reason why I personally would not choose wireheading, but other reasons have already been well discussed in this thread and I haven't seen anyone else zero in on this particular point.)
I've been thinking about wireheading and the nature of my values. Many people here have defended the importance of external referents or complex desires. My problem is, I can't understand these claims at all.
To clarify, I mean wireheading in the strict "collapsing into orgasmium" sense. A successful implementation would identify all the reward circuitry and directly stimulate it, or do something equivalent. It would essentially be a vastly improved heroin. A good argument for either keeping complex values (e.g. by requiring at least a personal matrix) or external referents (e.g. by showing that a simulation can never suffice) would work for me.
Also, I use "reward" as short-hand for any enjoyable feeling, as "pleasure" tends to be used for a specific one of them, among bliss, excitement and so on, and "it's not about feeling X, but X and Y" is still wireheading after all.
I tried collecting all related arguments I could find. (Roughly sorted from weak to very weak, as I understand them, plus link to example instances. I also searched any literature/other sites I could think of, but didn't find other (not blatantly incoherent) arguments.)
(There have also been technical arguments against specific implementations of wireheading. I'm not concerned with those, as long as they don't show impossibility.)
Overall, none of this sounds remotely plausible to me. Most of it is outright question-begging or relies on intuition pumps that don't even work for me.
It confuses me that others might be convinced by arguments of this sort, so it seems likely that I have a fundamental misunderstanding or there are implicit assumptions I don't see. I fear that I have a large inferential gap here, so please be explicit and assume I'm a Martian. I genuinely feel like Gamma in A Much Better Life.
To me, all this talk about "valueing something" sounds like someone talking about "feeling the presence of the Holy Ghost". I don't mean this in a derogatory way, but the pattern "sense something funny, therefore some very specific and otherwise unsupported claim" matches. How do you know it's not just, you know, indigestion?
What is this "valuing"? How do you know that something is a "value", terminal or not? How do you know what it's about? How would you know if you were mistaken? What about unconscious hypocrisy or confabulation? Where do these "values" come from (i.e. what process creates them)? Overall, it sounds to me like people are confusing their feelings about (predicted) states of the world with caring about states directly.
To me, it seems like it's all about anticipating and achieving rewards (and avoiding punishments, but for the sake of the wireheading argument, it's equivalent). I make predicitions about what actions will trigger rewards (or instrumentally help me pursue those actions) and then engage in them. If my prediction was wrong, I drop the activity and try something else. If I "wanted" something, but getting it didn't trigger a rewarding feeling, I wouldn't take that as evidence that I "value" the activity for its own sake. I'd assume I suck at predicting or was ripped off.
Can someone give a reason why wireheading would be bad?