Some arrangements of particles feel better than others. Why?
We have no general theories, only descriptive observations within the context of the vertebrate brain, about what produces pain and pleasure. It seems like there's a mystery here, a general principle to uncover.
Let's try to chart the mystery. I think we should, in theory, be able to answer the following questions:
(1) What are the necessary and sufficient properties for a thought to be pleasurable?
(2) What are the characteristic mathematics of a painful thought?
(3) If we wanted to create an artificial neural network-based mind (i.e., using neurons, but not slavishly patterned after a mammalian brain) that could experience bliss, what would the important design parameters be?
(4) If we wanted to create an AGI whose nominal reward signal coincided with visceral happiness -- how would we do that?
(5) If we wanted to ensure an uploaded mind could feel visceral pleasure of the same kind a non-uploaded mind can, how could we check that?
(6) If we wanted to fill the universe with computronium and maximize hedons, what algorithm would we run on it?
(7) If we met an alien life-form, how could we tell if it was suffering?
It seems to me these are all empirical questions that should have empirical answers. But we don't seem to have much for hand-holds which can give us a starting point.
Where would *you* start on answering these questions? Which ones are good questions, and which ones are aren't? And if you think certain questions aren't good, could you offer some you think are?
As suggested by shminux, here's some research I believe is indicative of the state of the literature (though this falls quite short of a full literature review):
Tononi's IIT seems relevant, though it only addresses consciousness and explicitly avoids valence. Max Tegmark has a formal generalization of IIT which he claims should apply to non-neural substrates. And although Tegmark doesn't address valence either, he posted a recent paper on arxiv noting that there *is* a mystery here, and that it seems topical for FAI research.
Current models of emotion based on brain architecture and neurochemicals (e.g., EMOCON) are somewhat relevant, though ultimately correlative or merely descriptive, and seem to have little universalization potential.
There's also a great deal of quality literature about specific correlates of pain and happiness- e.g., Building a neuroscience of pleasure and well-being and An fMRI-Based Neurologic Signature of Physical Pain. Luke covers Berridge's research in his post, The Neuroscience of Pleasure. Short version: 'liking', 'wanting', and 'learning' are all handled by different systems in the brain. Opioids within very small regions of the brain seem to induce the 'liking' response; elsewhere in the brain, opioids only produce 'wanting'. We don't know how or why yet. This sort of research constrains a general principle, but doesn't really hint toward one.
In short, there's plenty of research around the topic, but it's focused exclusively on humans/mammals/vertebrates: our evolved adaptations, our emotional systems, and our architectural quirks. Nothing on general or universal principles that would address any of (1)-(7). There is interesting information-theoretic / patternist work being done, but it's highly concentrated around consciousness research.
---
Bottom line: there seems to be a critically important general principle as to what makes certain arrangements of particles innately preferable to others, and we don't know what it is. Exciting!
First recommendation is to get to the bottom of what question you are actually asking. What are you actually trying to do? Do the right thing? Learn how to manipulate people? Learn how to torture? Become a pleasure delivery professional?
See disguised queries
It feels good? Some pretty heavy neuroscience to say anything beyond that. Again, what are you going to do with the answer to this question. Ask that question instead.
Also note that "necessary and sufficient" is an obsolete model of concepts. See the human's guide to words.
What does this mean? How do I calculate exactly how much pain someone will experience if I punch them? Again, ask the real question.
Um. Why would you want to do that? Is this simply a hypothetical to see if we understand the concept?
It really depends on what aspect you are interested in; you could create "pleasure" and "pain" by hacking up some kind of simple reinforcement learner, and I suppose you could shoehorn that into a neural network if you really wanted to. But why?
Note that a simple reinforcement learner "experiences" "pain" and "pleasure" in some sense, but not in the morally relevant sense. You will find that the moral aspect is much more anthropomorphic and much more complex, I think.
I guess you could have a little "visceral happiness" meter that gets filled up in the right conditions, but this would a profound waste of AGI capability, and probably doesn't do what you actually wanted. What is it you actually want?
Ask them? The same way we think we know for non-uploaded minds.
If I wanted to turn the universe into paperclips and meaningless crap, how would I do it? Why is your question interesting? Is this simply an excercise in learning how to fill the universe with X? You could pick a less confusing X.
I feel like you might be importing a few mistaken assumptions into this whole line of questioning. I recommend that you lurk more and read some of the stuff I linked.
Good question:
How would a potentially powerful optimizing process have to be constructed to be provably capable of steering towards some coherent objective(s) over the long run and through self-modifications?
Downvote preventers get downvoted.
Even if it turns out that there is no rigorously definable one-dimensional measure of valence we still need to search for physical correlates to pleasure and pain and find approximate measures to use when resolving moral dilemmas.
Regarding the response to (6), why don't you want to maximise hedons? Having a rigorous definition of what you are trying to maximise needn't mean that what you are trying to maximise is arbitrary to you, and that pleasure is complex (or maybe it is simple but we don't understand it yet) does not imply that we don't want it.