An indicator. If I can stick an electrode in your brain and keep you blissfully happy for the rest of your life, and hook you up to an intravenous drip to provide you with enough nutrients to keep you alive (but perfectly useless to everyone else, and perfectly incapable of achieving any of the goals you might have had before you were hooked up) for, say, 25 years -- would you want it done?
Ah, wireheading. No. For reasons that I'll get into below.
I would not, because I care about things other than happiness. Happiness is my brain's way of indicating that some kinds of goals are being met, and it's a useful signal of those things, but it's no more perfectly reliable than any number of other internal behaviour-guiding signals in my brain. If I trusted them all completely, I would make a lot of poor decisions about what to eat and when, whom to have sex with and when, when to give up on something because it's too hard, etc., etc., etc.
What would make those decisions poor, however? Is it because they would reduce your long-term happiness?
You're engaging in a bit of mind-body duality here; your brain isn't a separate thing that imposes stuff on your mind, your brain is your mind. Your brain's happiness is your happiness; and likewise, the reverse. First, it's important to note that pleasure isn't happiness; this is something I assume you're consciously aware of, but you appear to conflate the two somewhat in terms of what kinds of decisions you use as examples.
Why wouldn't I wirehead? Because the idea horrifies me. As soon as the electrodes go in, I'm quite certain I'd prefer that state. I wouldn't even agree to five seconds of it, however, because I expect after five seconds, I'd readily agree to twenty five years; a few seconds of wireheading is equivalent to twenty five years, because I expect it would erode any desire not to do it. And the extreme unhappiness I feel with relation to the idea of wireheading tells me that it isn't right for me, and, I expect, it isn't right for people in general.
I call happiness the indicator; if you rewire my turn signal indicators in my dashboard, which normally indicate whether or not my turn signals are flashing, so that they are always on, are my turn signals always on? Or have you merely destroyed a useful signal?
What would make those decisions poor, however? Is it because they would reduce your long-term happiness?
No. (Not entirely, anyway; happiness is one of my goals.)
You're engaging in a bit of mind-body duality here
I really don't think I am. I did not claim (and I do not believe) that those things are being signalled by/in my brain to something outside my brain. If I say "the contents of address 0x1453C are the chess program's estimate of how much danger the white king is in", I'm not claiming it's using the contents of that address to signal ...
I grew up in an atheistic household.
Almost needless to say, I was relatively hostile towards religion for most of my early life. A few things changed that.
First, the apology of a pastor. A friend of mine was proselytizing at me, and apparently discussed it with his pastor; the pastor apologized to my parents, and explained to my friend he shouldn't be trying to convert people. My friend apologized to me after considering the matter. We stayed friends for a little while afterwards, although I left that school, and we lost contact.
I think that was around the time that I realized that religion is, in addition to being a belief system, a way of life, and not necessarily a bad one.
The next was actually South Park's Mormonism episode, which pointed out that a belief system could be desirable on the merits of the way of life it represented, even if the beliefs themselves are stupid. This tied into Douglas Adam's comment on Feng Shui, that "...if you disregard for a moment the explanation that's actually offered for it, it may be there is something interesting going on" - which is to say, the explanation for the belief is not necessarily the -reason- for the belief, and that stupid beliefs may actually have something useful to offer - which then requires us to ask whether the beliefs are, in fact, stupid.
Which is to say, beliefs may be epistemically irrational while being instrumentally rational.
The next peace I made with belief actually came from quantum physics, and reading about how there were several disparate and apparently contradictory mathematical systems, which all predicted the same thing. It later transpired that they could all be generalized into the same mathematical system, but I hadn't read that far before the isomorphic nature of truth occurred to me; you can have multiple contradictory interpretations of the same evidence that all predict the same thing.
Up to this point, however, I still regarded beliefs as irrational, at least on an epistemological basis.
The next peace came from experiences living in a house that would have convinced most people that ghosts are real, which I have previously written about here. I think there are probably good explanations for every individual experience even if I don't know them, but am still somewhat flummoxed by the fact that almost all the bizarre experiences of my life all revolve around the same physical location. I don't know if I would accept money to live in that house again, which I guess means that I wouldn't put money on the bet that there wasn't something fundamentally odd about the house itself - a quality of the house which I think the term "haunted" accurately conveys, even if its implications are incorrect.
If an AI in a first person shooter dies every time it walks into a green room, and experiences great disutility for death, how many times must it walk into a green room before it decides not to do that anymore? I'm reasonably confident on a rational level that there was nothing inherently unnatural about that house, nothing beyond explanation, but I still won't "walk into the green room."
That was the point at which I concluded that beliefs can be -rational-. Disregard for a moment the explanation that's actually offered for them, and just accept the notion that there may be something interesting going on underneath the surface.
If we were to hold scientific beliefs to the same standard we hold religious beliefs - holding the explanation responsible rather than the predictions - scientific beliefs really don't come off looking that good. The sun isn't the center of the universe; some have called this theory "less wrong" than an earth-centric model of the universe, but that's because the -predictions- are better; the explanation itself is still completely, 100% wrong.
Likewise, if we hold religious beliefs to the same standard we hold scientific beliefs - holding the predictions responsible rather than the explanations - religious beliefs might just come off better than we'd expect.