For example, when a rat in a Skinner box is hungry (ie its satiety variable has deviated in the direction of hunger), and then it presses a lever and gets a food pellet and its satiety variable goes back to its reference range, would PCTists explain that as getting rewarded for pressing the lever and expect it to press the lever again next time it's hungry?
The PCT learning model doesn't require reinforcement at the control level, as its model of memory is a mapping from reference levels to predicted levels of other variables. I.e., when the rat notices that the lever-pressing is paired with food, a link is made between two perceptual variables: the position of the lever, and the availability of food. This means that the rat can learn that food is available, even when it's not hungry.
Where reinforcement is relevant to PCT is in the strength of the linkage and in the likelihood of its being recorded. If the rat is hungry, then the linkage is more salient, and more likely to be learned.
Notice though, that again the animal's internal state is of primary importance, not the stimulus/response. In a sense, you could say that you can teach an animal that a stimulus and response are paired, but this isn't the same as making the animal behave. If we starved you and made you press a lever for your food, you might do it, or you might tell us to fork off. Yet, we don't claim that you haven't learned that pressing the lever leads to food in that case.
(As Richard says, it's well established that you can torture living creatures until they accede to your demands, but it won't necessarily tell you much about how the creature normally works.)
In any case, PCT allows for the possibility of learning without "reinforcement" in the behaviorist sense, unless you torture the definition of reinforcement to the point that anything is reinforcement.
Regarding the leptin/ghrelin question, my understanding is that PCT as a psych-physical model primarily addresses those perceptual variables that are modeled by neural analog -- i.e., an analog level maintained in a neural delay loop. While Powers makes many references to other sorts of negative feedback loops in various organisms from cats to E. coli, the main thrust of his initial book deals with building up a model of what's going on, feedback-loopwise, in the nervous system and brain, not the body's endocrine systems.
To put it another way, PCT doesn't say that control systems are universal, only that they are ubiquitous, and that the bulk of organisms' neural systems are assembled from a relatively small number of distinct component types that closely resemble the sort of components that humans use when building machinery.
IOW, we should not expect that PCT's model of neural control systems would be directly applicable to a hormone level issue. However, we can reason from general principles and say that one difference between a PCT model of the leptin/ghrelin question is that PCT includes an explicit model of hierarchy and conflict in control networks, so that we can answer questions about what happens if both leptin and ghrelin are present (for example).
If those signals are at the same level of control hierarchy, we can expect conflict to result in oscillation, where the system alternates between trying to satisfy one or the other. Or, if they're at different levels of hierarchy, then we can expect one to override the other.
But, unlike a behavioral model where the question of precedence between different stimuli and contexts is open to interpretation, PCT makes some testable predictions about what actually constitutes hierarchy, both in terms of expected behavior, and in terms of the physical structure of the underlying control circuitry.
That is, if you could dissect an organism and find the neurons, PCT predicts a certain type of wiring to exist, i.e., that a dominant controller will have wiring to set the reference levels for lower-level controllers, but not vice-versa.
Second, PCT predicts that a dominant perception must be measured at a longer time scale than a dominated one. That is, the lower-level perception must have a higher sampling rate than the higher-level perception. Thus, for example, as a rat becomes hungrier (a longer-term perceptual variable), its likelihood of pressing a lever to receive food in spite of a shock is increased.
AFAICT, behaviorism can "explain" results like these, but does not actually predict them, in the sense that PCT is spelling out implementation-level details that behaviorism leaves to hand-waving. IOW, PCT is considerably more falsifiable than behaviorism, at least in principle. Eventually, PCT's remaining predictions (i.e., the ones that haven't already panned out at the anatomical level) will either be proven or disproven, while behaviorism doesn't really make anatomical predictions about these matters.
Imagine a robot with a turret-mounted camera and laser. Each moment, it is programmed to move forward a certain distance and perform a sweep with its camera. As it sweeps, the robot continuously analyzes the average RGB value of the pixels in the camera image; if the blue component passes a certain threshold, the robot stops, fires its laser at the part of the world corresponding to the blue area in the camera image, and then continues on its way.
Watching the robot's behavior, we would conclude that this is a robot that destroys blue objects. Maybe it is a surgical robot that destroys cancer cells marked by a blue dye; maybe it was built by the Department of Homeland Security to fight a group of terrorists who wear blue uniforms. Whatever. The point is that we would analyze this robot in terms of its goals, and in those terms we would be tempted to call this robot a blue-minimizer: a machine that exists solely to reduce the amount of blue objects in the world.
Suppose the robot had human level intelligence in some side module, but no access to its own source code; that it could learn about itself only through observing its own actions. The robot might come to the same conclusions we did: that it is a blue-minimizer, set upon a holy quest to rid the world of the scourge of blue objects.
But now stick the robot in a room with a hologram projector. The hologram projector (which is itself gray) projects a hologram of a blue object five meters in front of it. The robot's camera detects the projector, but its RGB value is harmless and the robot does not fire. Then the robot's camera detects the blue hologram and zaps it. We arrange for the robot to enter this room several times, and each time it ignores the projector and zaps the hologram, without effect.
Here the robot is failing at its goal of being a blue-minimizer. The right way to reduce the amount of blue in the universe is to destroy the projector; instead its beams flit harmlessly through the hologram.
Again, give the robot human level intelligence. Teach it exactly what a hologram projector is and how it works. Now what happens? Exactly the same thing - the robot executes its code, which says to scan the room until its camera registers blue, then shoot its laser.
In fact, there are many ways to subvert this robot. What if we put a lens over its camera which inverts the image, so that white appears as black, red as green, blue as yellow, and so on? The robot will not shoot us with its laser to prevent such a violation (unless we happen to be wearing blue clothes when we approach) - its entire program was detailed in the first paragraph, and there's nothing about resisting lens alterations. Nor will the robot correct itself and shoot only at objects that appear yellow - its entire program was detailed in the first paragraph, and there's nothing about correcting its program for new lenses. The robot will continue to zap objects that register a blue RGB value; but now it'll be shooting at anything that is yellow.
The human-level intelligence version of the robot will notice its vision has been inverted. It will know it is shooting yellow objects. It will know it is failing at its original goal of blue-minimization. And maybe if it had previously decided it was on a holy quest to rid the world of blue, it will be deeply horrified and ashamed of its actions. It will wonder why it has suddenly started to deviate from this quest, and why it just can't work up the will to destroy blue objects anymore.
The robot goes to Quirinus Quirrell, who explains that robots don't really care about minimizing the color blue. They only care about status and power, and pretend to care about minimizing blue in order to impress potential allies.
The robot goes to Robin Hanson, who explains that there are really multiple agents within the robot. One of them wants to minimize the color blue, the other wants to minimize the color yellow. Maybe the two of them can make peace, and agree to minimize yellow one day and blue the next?
The robot goes to Anna Salamon, who explains that robots are not automatically strategic, and that if it wants to achieve its goal it will have to learn special techniques to keep focus on it.
I think all of these explanations hold part of the puzzle, but that the most fundamental explanation is that the mistake began as soon as we started calling it a "blue-minimizing robot". This is not because its utility function doesn't exactly correspond to blue-minimization: even if we try to assign it a ponderous function like "minimize the color represented as blue within your current visual system, except in the case of holograms" it will be a case of overfitting a curve. The robot is not maximizing or minimizing anything. It does exactly what it says in its program: find something that appears blue and shoot it with a laser. If its human handlers (or itself) want to interpret that as goal directed behavior, well, that's their problem.
It may be that the robot was created to achieve a specific goal. It may be that the Department of Homeland Security programmed it to attack blue-uniformed terrorists who had no access to hologram projectors or inversion lenses. But to assign the goal of "blue minimization" to the robot is a confusion of levels: this was a goal of the Department of Homeland Security, which became a lost purpose as soon as it was represented in the form of code.
The robot is a behavior-executor, not a utility-maximizer.
In the rest of this sequence, I want to expand upon this idea. I'll start by discussing some of the foundations of behaviorism, one of the earliest theories to treat people as behavior-executors. I'll go into some of the implications for the "easy problem" of consciousness and philosophy of mind. I'll very briefly discuss the philosophical debate around eliminativism and a few eliminativist schools. Then I'll go into why we feel like we have goals and preferences and what to do about them.