LESSWRONG
LW

Comment Permalink

The robot is not consequentialist, its decisions are not controlled by the dependence of facts about the future on its decisions.

Wei Dai13y210

Good point, but the fact that humans are consequentialists (at least partly) doesn't seem to make the problem much easier. Suppose we replace Yvain's blue-minimizer robot with a simple consequentialist robot that has the same behavior (let's say it models the world as a 2D grid of cells that have intrinsic color, it always predicts that any blue cell that it shoots at will turn some other color, and its utility function assigns negative utility to the existence of blue cells). What does this robot "actually want", given that the world is not really a 2D grid of cells that have intrinsic color?

See in context

326 The Blue-Minimizing Robot

by Scott Alexander

4th Jul 2011

4 min read

160

326

Imagine a robot with a turret-mounted camera and laser. Each moment, it is programmed to move forward a certain distance and perform a sweep with its camera. As it sweeps, the robot continuously analyzes the average RGB value of the pixels in the camera image; if the blue component passes a certain threshold, the robot stops, fires its laser at the part of the world corresponding to the blue area in the camera image, and then continues on its way.

Watching the robot's behavior, we would conclude that this is a robot that destroys blue objects. Maybe it is a surgical robot that destroys cancer cells marked by a blue dye; maybe it was built by the Department of Homeland Security to fight a group of terrorists who wear blue uniforms. Whatever. The point is that we would analyze this robot in terms of its goals, and in those terms we would be tempted to call this robot a blue-minimizer: a machine that exists solely to reduce the amount of blue objects in the world.

Suppose the robot had human level intelligence in some side module, but no access to its own source code; that it could learn about itself only through observing its own actions. The robot might come to the same conclusions we did: that it is a blue-minimizer, set upon a holy quest to rid the world of the scourge of blue objects.

But now stick the robot in a room with a hologram projector. The hologram projector (which is itself gray) projects a hologram of a blue object five meters in front of it. The robot's camera detects the projector, but its RGB value is harmless and the robot does not fire. Then the robot's camera detects the blue hologram and zaps it. We arrange for the robot to enter this room several times, and each time it ignores the projector and zaps the hologram, without effect.

Here the robot is failing at its goal of being a blue-minimizer. The right way to reduce the amount of blue in the universe is to destroy the projector; instead its beams flit harmlessly through the hologram.

Again, give the robot human level intelligence. Teach it exactly what a hologram projector is and how it works. Now what happens? Exactly the same thing - the robot executes its code, which says to scan the room until its camera registers blue, then shoot its laser.

In fact, there are many ways to subvert this robot. What if we put a lens over its camera which inverts the image, so that white appears as black, red as green, blue as yellow, and so on? The robot will not shoot us with its laser to prevent such a violation (unless we happen to be wearing blue clothes when we approach) - its entire program was detailed in the first paragraph, and there's nothing about resisting lens alterations. Nor will the robot correct itself and shoot only at objects that appear yellow - its entire program was detailed in the first paragraph, and there's nothing about correcting its program for new lenses. The robot will continue to zap objects that register a blue RGB value; but now it'll be shooting at anything that is yellow.

The human-level intelligence version of the robot will notice its vision has been inverted. It will know it is shooting yellow objects. It will know it is failing at its original goal of blue-minimization. And maybe if it had previously decided it was on a holy quest to rid the world of blue, it will be deeply horrified and ashamed of its actions. It will wonder why it has suddenly started to deviate from this quest, and why it just can't work up the will to destroy blue objects anymore.

The robot goes to Quirinus Quirrell, who explains that robots don't really care about minimizing the color blue. They only care about status and power, and pretend to care about minimizing blue in order to impress potential allies.

The robot goes to Robin Hanson, who explains that there are really multiple agents within the robot. One of them wants to minimize the color blue, the other wants to minimize the color yellow. Maybe the two of them can make peace, and agree to minimize yellow one day and blue the next?

The robot goes to Anna Salamon, who explains that robots are not automatically strategic, and that if it wants to achieve its goal it will have to learn special techniques to keep focus on it.

I think all of these explanations hold part of the puzzle, but that the most fundamental explanation is that the mistake began as soon as we started calling it a "blue-minimizing robot". This is not because its utility function doesn't exactly correspond to blue-minimization: even if we try to assign it a ponderous function like "minimize the color represented as blue within your current visual system, except in the case of holograms" it will be a case of overfitting a curve. The robot is not maximizing or minimizing anything. It does exactly what it says in its program: find something that appears blue and shoot it with a laser. If its human handlers (or itself) want to interpret that as goal directed behavior, well, that's their problem.

It may be that the robot was created to achieve a specific goal. It may be that the Department of Homeland Security programmed it to attack blue-uniformed terrorists who had no access to hologram projectors or inversion lenses. But to assign the goal of "blue minimization" to the robot is a confusion of levels: this was a goal of the Department of Homeland Security, which became a lost purpose as soon as it was represented in the form of code.

The robot is a behavior-executor, not a utility-maximizer.

In the rest of this sequence, I want to expand upon this idea. I'll start by discussing some of the foundations of behaviorism, one of the earliest theories to treat people as behavior-executors. I'll go into some of the implications for the "easy problem" of consciousness and philosophy of mind. I'll very briefly discuss the philosophical debate around eliminativism and a few eliminativist schools. Then I'll go into why we feel like we have goals and preferences and what to do about them.

Ontological CrisisMotivations

Frontpage

326

Mentioned in

133Secrets of the eliminati

126Urges vs. Goals: The analogy to anticipation and belief

126The Library of Scott Alexandria

91An artificially structured argument for expecting AGI ruin

81Bridge Collapse: Reductionism as Engineering Problem

Load More (5/19)

The Blue-Minimizing Robot

New Comment

161 comments, sorted by

top scoring

Click to highlight new comments since: Today at 9:58 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Vladimir_Nesov13y440

The robot is not consequentialist, its decisions are not controlled by the dependence of facts about the future on its decisions.

[-]Wei Dai13y210

[-]steven046113y310

What does this robot "actually want", given that the world is not really a 2D grid of cells that have intrinsic color?

Who cares about the question what the robot "actually wants"? Certainly not the robot. Humans care about the question what they "actually want", but that's because they have additional structure that this robot lacks. But with humans, you're not limited to just looking at what they do on auto-pilot; instead, you can just ask the aforementioned structure when you run into problems like this. For example, if you asked me what I really wanted under some weird ontology change, I could say, "I have some guesses, but I don't really know; I would like to defer to a smarter version of me". That's how I understand preference extrapolation: not as something that looks at what your behavior suggests that you're trying to do and then does it better, but as something that poses the question of what you want to some system you'd like to answer the question for you.

It looks to me like there's a mistaken tendency among many people here, including some very smart people, to say that I'd be irrational to let my stated preferences deviate fr... (read more)

[-]Wei Dai13y210

In other words, our "actual values" come from our being philosophers, not our being consequentialists.

It seems plausible to me, and I'm not sure that "many" others do disagree with you.

5cousin_it13y

That would imply a great diversity of value systems, because philosophical intuitions differ much more from person to person than primitive desires. Some of these value systems (maybe including yours) would be simple, some wouldn't. For example, my "philosophical" values seem to give large weight to my "primitive" values.

4[anonymous]13y

That might be a procedure that generates human preference, but it is not a general preference extrapolation procedure. E.g suppose we replace Wei Dai's simple consequentialist robot with a robot that has similar behavior, but that also responds to the question, "What system do you want to answer the question of what you want for you?" with the answer, "A version of myself better able to answer that question. Maybe it should be smarter and know more things and be nicer to strangers and not have scope insensitivity and be less prone to skipping over invisible moral frameworks and have conecepts that are better defined over attribute space and be automatically strategic and super commited and stuff like that? But since I'm not that smart and I pass over moral frameworks and stuff, eveything I just said is probably insufficient to specify the right thing. Maybe you can look at my source code and figure out what I mean by right and then do the thing that a person who better understood that would do?" And then goes right back to zapping blue.

1[anonymous]13y

Suppose we replace Wei Dai's simple consequentialist robot with a robot that has similar behavior, but that also responds to the question, "What system do you want to answer the question of what you want for you?" with the answer, "I want to decide for myself" and responds to the question, "What do you want to do?" with the answer, "I want to make babies happy. Oh and help grandmother out of the burning building. Oh, and without killing her. Oh, and to preserve complex novelty. Oh and boredom. Oh, and there should still be people in the world who are trying to improve it. Oh and...damnit, this is complicated. Okay, never mind, I want you to ask the version of myself who I presently think is smart enough to answer this question and who knows what the right thing to do is even better than me." It can answer those two questions, but if you ask it to clarify the last response, it just blows up.

8Vladimir_Nesov13y

Actually, this notion of consequentialism gives a new and the only clue I know of about how to infer agent goals, or how to constrain the kinds of considerations that should be considered goals, as compared to the other stuff that moves your action incidentally, such as psychological drives or laws of physics. I wonder if Eliezer had this insight before, given that he wrote a similar comment to this thread. I wasn't ready to see this idea on my own until a few weeks ago, and this thread is the first time I thought about the question given the new framework, and saw the now-obvious construction. This deserves more than a comment, so I'll be working on a two-post sequence to write this up intelligibly. Or maybe it's actually just stupid, I'll try to figure that out. ---------------------------------------- (A summary from my notes, in case I get run over by a bus; this uses a notion of "dependence" for which a toy example is described in my post on ADT, but which is much more general: ) The idea of consequentialism, of goal-directed control, can be modeled as follows. If a fact A is controlled by (can be explained/predicted based on) a dependence F: A->O, then we say that A is a decision (action) driven by a consequentialist consideration F, which in turn looks at how A controls the morally relevant fact O. For a given decision A, there could be many different morally relevant facts O such that the dependence A->O has explanatory power about A. The more about A can a dependence A->O explain, the more morally relevant O is. Finding highly relevant facts O essentially captures A's goals. This model has two good properties. First, logical omniscience (in particular, just knowledge of actual action) renders the construction unusable, since we need to see dependencies A->O as ambient concepts explaining A, so both A and A->O need to remain potentially unknown. (This is the confusing part. It also lends motivation to the study of complete collection of moral arguments

3lukeprog13y

I very much look forward to your short sequence on this. I hope you will also explain your notion of dependence in detail.

1Vladimir_Nesov13y

For the record, I mostly completed a draft of a prerequisite post (first of the two I had in mind) a couple of weeks ago, and it's just no good, not much better than what one would take away from reading the previously published posts, and not particularly helpful in clarifying the intuition expressed in the above comments. So I'm focusing on improving my math skills, which I expect will help with formalization/communication problem (given a few months), as well as with moving forward. I might post some version of the post, but it seems it won't be able to serve the previously intended purpose.

0lukeprog13y

Bummer. As for communication, it would help me (at least) if you used words in their normal senses unless they are a standard LW term of art (e.g. 'rationalist' means LW rationalist not Cartesian rationalist ) or unless you specify that you're using the term in an uncommon sense.

1Vladimir_Nesov13y

Don't see how this is related to this thread, and correspondingly what kinds of word misuse you have in mind.

0lukeprog13y

It isn't related to this thread. I was thinking of past confusions between us over 'metaethics' and 'motivation' and 'meaning' where I didn't realize until pretty far into the discussion that you were using these terms to mean something different than they normally mean. I'd generally like to avoid that kind of thing; that's all I meant.

0Vladimir_Nesov13y

Well, I'm mostly not interested in the concepts corresponding to how these words are "normally" used. The miscommunication problems resulted from both me misinterpreting your usage, and your misinterpreting my usage. I won't be misinterpreting your usage in similar cases in the future, as I now know better what you mean by which words, and in my own usage, as we discussed a couple of times, I'll be more clear through using disambiguating qualifiers, which in most cases amounts to writing word "normative" more frequently. (Still unclear/strange why you brought it up in this particular context, but no matter...)

0lukeprog13y

Yup, that sounds good. I brought it up because you mentioned communication, and your comment showed up in my LW inbox today.

6twanvl13y

The robot wants to minimize the amount of blue it sees in its grid representation of the world. It can do this by affecting the world with a laser. But it could also change its camera system so that it sees less blue. If there is no term in the utility function that says that the grid has to model reality, then both approaches are equally valid.

5cousin_it10y

steven0461's comment notwithstanding, I can take a guess at what the robot actually wants. I think it wants to take the action that will minimize the number of blue cells existing in the world, according to the robot's current model of the world. That rule for choosing actions probably doesn't correspond to any coherent utility function over the real world, but that's not really a surprise. The interesting question that you probably meant to ask is whether the robot's utility function over its model of the world can be converted to a utility function over the real world. But the robot won't agree to any such upgrade, so the question is kinda moot. That might sound hopeless for CEV, but fortunately humans aren't consequentialists with a fixed model of the world. Instead they seem to be motivated by pleasure and pain, which you can't disprove out of existence by coming up with a better model. So maybe there's hope in that direction.

5Armok_GoB13y

To avoid SEEING blue things. If the model is good enough for it it'd search out a mirror and laser it's own camera so that it could NEVER see a blue pixel again. This can be modelled using human empathy by equating the sensation of seeing blue with pain. You don't care to minimize damage to your body (if it's not somehting that actually cripples you), but you care about not getting the signal about it happening, and your reaction to a pill that turned you masochist would be very different than your reaction to a murder pill. Edit: huh? I am surprised that this is downvoted, and the most probable reason is that I'm wrong in some obvious way that I can't see, can someone please tell me how? (Or maybe my usage of empathy was interpreted way to literally. )

2Vladimir_Nesov13y

(Reading this comment first might be helpful.) To answer your thought experiment. It doesn't matter what the agent thinks it's acting based on, we look at it from the outside instead (but using a particular definition/dependence that specifies the agent), and ask how its action depends on the dependence of the actual future on its actual action. Agent's misconceptions don't enter this question. If the misconceptions are great, it'll turn out that the dependence of actual future on agent's action doesn't control its action, or controls it in some unexpected way. Alternatively, we could say that it's not the actual future that is morally relevant for the agent, but some other strange fact, in which case the agent could be said to be optimizing a world that is not ours. From yet another perspective, the role of the action could be played by something else, but then it's not clear why we are considering such a model and talking about this particular actual agent at the same time.

0[anonymous]13y

Is that something you can see from the outside? If I argmax over actions in expected-paper-clips or over updateless-prior-expected-paper-clips, how can you translate my black box behavior over possible worlds into the dependence of my behavior on the dependence of the worlds on my behavior?

0Vladimir_Nesov13y

See the section "Utility functions" of this post: it shows how a dependence between two fixed facts could be restored in an ideal case where we can learn everything there is to learn about it. Similarly, you could consider the fact of which dependence holds between two facts, with various specific functions as its possible values, and ask what can you infer about that other fact if you assume that the dependence is given by a certain function. More generally, a dependence follows possible inferences, things that could be inferred about one fact if you learn new things about the other fact. It needs to follow all of such inferences, to the best of agent's ability, otherwise it won't be right and you'll get incorrect decisions (counterfactual models).

2Vladimir_Nesov13y

Edit: Actually, never mind, I missed your point. Will reply again later. (Done.) Our world is not really a 2D grid, its world could be. It won't be a consequentialist about our world then, for that requires the dependence of its decisions on the dependence of our world on its decisions. It looks like the robot you described wants to minimize the 2D grid greenness, and would do that sensibly in the context of 2D grid world, or any world that can influence the 2D grid world. For the robot, our world doesn't exist in the same sense as 2D grid world doesn't exist for us, even though we could build an instance of the robot in our world. If we do build such an instance, the robot, if extremely rational and not just acting on heuristics adapted for its natural habitat, could devote its our-worldly existence to finding ways of acausally controlling its 2D world. For example, if there are some 2D-worlders out there simulating our world, it could signal to them something that is expected to reduce greenness. (This all depends on the details of robot's decision-making tools, of course. It could really be talking about our world, but then its values collapse, and it could turn out to not be a consequentialist after all, or optimizing something very different, to the extent the conflict in the definitions is strong.)

[-][anonymous]13y370

The conclusion I'd draw from this essay is that one can't necessarily derive a "goal" or a "utility function" from all possible behavior patterns. If you ask "What is the robot's goal?", the answer is, "it doesn't have one," because it doesn't assign a total preference ordering to states of the world. At best, you could say that it prefers state [I SEE BLUE AND I SHOOT] to state [I SEE BLUE AND I DON'T SHOOT]. But that's all.

This has some implications for AI, I think. First of all, not every computer program has a goal or a utility function. There is no danger that your TurboTax software will take over the world and destroy all human life, because it doesn't have a general goal to maximize the number of completed tax forms. Even rather sophisticated algorithms can completely lack goals of this kind -- they aren't designed to maximize some variable over all possible states of the universe. It seems that the narrative of unfriendly AI is only a risk if an AI were to have a true goal function, and many useful advances in artificial intelligence (defined in the broad sense) carry no risk of this kind.

Do humans have goals? I don't know; it... (read more)

8Gurkenglas11y

No; placing a blue-tinted mirror in front of him will have him shoot himself even though that greatly diminishes his future ability to shoot. Generally a generic program really can't be assigned any nontrivial utility function.

2Yaacov9y

Destroying the robot greatly diminishes its future ability to shoot, but it would also greatly diminishes its future ability to see blue. The robot doesn't prefer 'shooting blue' to 'not shooting blue', it prefers 'seeing blue and shooting' to 'seeing blue and not shooting'. So the original poster was right. Edit: I'm wrong, see below

3Gurkenglas9y

If the robot knows that its camera is indestructible but its gun isn't, it would still shoot at the mirror and destroy only its gun.

1Vanilla_cabs5y

So it would be [I SEE BLUE AND I TRY TO SHOOT]. ... except that it wouldn't mind if shooting itself damaged its own program so that it wouldn't even try to shoot if it saw blue anymore. Ok, I am inclined to agree that its behaviour can't be described in terms of goals.

4Raw_Power13y

This is a very awesome post. Thumbs up.

2JamesAndrix13y

What does it mean for a program to have intelligence if it does not have a goal? (or have components that have goals) The point of any incremental intelligence increase is to let the program make more choices, and perhaps choices at higher levels of abstraction. Even at low intelligence levels, the AI will only 'do a good job' if the basis of those choices adequately matches the basis we would use to make the same choice. (a close match at some level of abstraction below the choice, not the substrate and not basic algorithms) Creating 'goal-less' AI still has the machine making more choices for more complex reasons, and allows for non-obvious mismatches between what it does and what we intended it to do. Yes, you can look at paperclip-manufacturing software and see that it is not a paper-clipper, but some component might still be optimizing for something else entirely. We can reject the anthropomorphically obvious goal and there can still be an powerful optimization process that affects the total system, at the expense of both human values and produced paperclips.

2asr13y

Consider automatic document translation. Making the translator more complex and more accurate doesn't imbue it with goals. It might easily be the case that in a few years, we achieve near-human accuracy at automatic document translation without major breakthroughs in any other area of AI research.

1JamesAndrix13y

Making it more accurate is not the same as making it more intelligent. The question is: How does making something "more intelligent" change the nature of the inaccuracies? In translation especially there can be a bias without any real inaccuracy . Goallessness at the level of the program is not what makes translators safe. They are safe because neither they nor any component is intelligent.

3asr13y

Most professional computer scientists and programmers I know routinely talk about "smart", "dumb", "intelligent" etc algorithms. In context, a smarter algorithm exploits more properties of the input or the problem. I think this is a reasonable use of language, and it's the one I had in mind. (I am open to using some other definition of algorithmic intelligence, if you care to supply one.) I don't see why making an algorithm smarter or more general would make it dangerous, so long as it stays fundamentally a (non-self-modifying) translation algorithm. There certainly will be biases in a smart algorithm. But dumb algorithms and humans have biases too.

1JamesAndrix13y

I generally go with cross domain optimization power. http://wiki.lesswrong.com/wiki/Optimization_process Note that optimization target is not the same thing as a goal, and the process doesn't need to exist within obvious boundaries. Evolution is goalless and disembodied. If an algorithm is smart because a programmer has encoded everything that needs to be known to solve a problem, great. That probably reduces potential for error, especially in well-defined environments. This is not what's going on in translation programs, or even the voting system here. (based on reddit) As systems like this creep up in complexity, their errors and biases become more subtle. (especially since we 'fix' them so that they usually work well) If an algorithm happens to be powerful in multiple domains, then the errors themselves might be optimized for something entirely different, and perhaps unrecognizable. By your definition I would tend to agree that they are not dangerous, so long as their generalized capabilities are below human level, (seems to be the case for everything so far) with some complex caveats. For example 'non-self-modifying' is a likely false sense of security. If an AI has access to a medium which can be used to do computations, and the AI is good at making algorithms, then it could (Edit: It could build a powerful if not superintelligent program.) Also, my concern in this thread has never been about the translation algorithm, the tax program, or even the paperclipper. It's about some sub-process which happens to be a powerful optimizer. (in a hypothetical situation where we do more AI research on the premise that it is safe if it is in a goalless program.

0Nebu9y

This is a very interesting question, thanks for making me think about it. (Based on your other comments elsewhere in this thread), it seems like you and I are in agreement that intelligence is about having the capability to make better choices. That is, two agents given an identical problem and identical resources to work with, the agent that is more intelligent is more likely to make the "better" choice. What does "better" mean here? We need to define some sort of goal and then compare the outcome of their choices and how closely those outcome matches those goals. I have a couple of disorganized thoughts here: * The goal is just necessary for us, outsiders, to compare the intelligence of the two agents. The goal is not necessary for the existence of intelligence in the agents if no one's interested in measuring their intelligence. * Assuming the agents are cooperative, you can temporarily assign subgoals. For example, perhaps you and I would like to know which one of us is smarter. You and I might have many different goals, but we might agree to temporarily take on a similar goal (e.g. win this game of chess, or get the highest amount of correct answers on this IQ test, etc.) so that our intelligence can be compared. * The "assigning" of goals to an intelligence strongly implies to me that goals are orthogonal to intelligence. Intelligence is the capability to fulfil any general goal, and it's possible for someone to be intelligent even if they do not (currently, or ever) have any goals. If we come up with a new trait called Sodadrinkability which is the capability to drink a given soda, one can say that I possess Sodadrinkability -- that I am capable of drinking a wide range of possible sodas provided to me -- even if I do not currently (or ever) have any sodas to drink.

0dfpolis10y

Let me suggest that the difference between goal-less behavior and goal-driven behavior, is that goal-driven behavior seeks means to attain its end. The means will vary with circumstances, while the end remains relatively invariant. Another indication of goal-driven behavior is that means are often prepared in anticipation of need, rather than in response to present need. I said "relatively invariant" because goals can be and often are heirarical. An example was outlined by Maslow in his "A Theory of Human Motivation" in the Psychological Review (1943). Maslow aside, in problem solving, we often resort to staged solutions in which the means to a higher order goal become a new sub-goal and so on iteratively -- until we reach low level goals within our immediate grasp. A second point is that terms such as "purposeful" and "goal-seeking" are analogously predicated. To be analogousley predicated, a term is applied to differnt cases with a meaning that is partly the same and partly different. Thus, a goal-seeking robot is not goal-seeking because it intends any goals of its own, but because it is the vehicle by which designer seeks to effect his or her goals. In the parable, if the goal was the destruction of blue uniformed enemies, that goal was only intended by the robots creators. Since the robot is an instantiated means of attaining that gosl, we may speak, analogously, of it as having the same goal. The important point is that we mean different things in saying "the designer has a goal" and "the robot has a goal." Each works toward same end (so the meaning iis partly the same), but only the designer intends that end (so the meaning iis partly different). (BTW this kind of analogy is an "analogy of attribution.") The fact that the robot is ineffective in attaining its end is a side issue that might be solved by employing better algoritms (edge and pattern recognition, etc.) There is no evidence that better algorithms will give the robot intentions in the sense that

[-]Quirinus_Quirrell13y340

Also, you misspelled my name - it's Quirinus, not Quirinius.

4John_Maxwell10y

Hijacking top comment... To finish reading Yvain's sequence, check out the corresponding sequence page).

[-]Richard_Kennaway13y330

I'll be interested to see where you go with this, but it seems to me that saying, "look, this is the program the robot runs, therefore it doesn't really have a goal", is exactly like saying "look, it's made of atoms, therefore it doesn't really have a goal".

Goals are precisely explained (like rainbows), and not explained away (like kobolds), as the controlled variables of control systems. This robot is such a system. The hypothetical goal of its designers at the Department of Homeland Security is also a goal. That does not make the robot's goal not a goal; it just makes it a different goal.

We feel like we have goals and preferences because we do, in fact, have goals and preferences, and we not only have them, but we are also aware of having them. The robot is not aware of having the goal that it has. It merely has it.

[-]Scott Alexander13y350

First of all, your control theory work was...not exactly what started me thinking along these lines, but what made it click when I realized the lines I had been thinking along were similar to the ones I had read about in one of your introductory posts about performing complex behaviors without representations. So thank you.

Second - When you say the robot has a "different goal", I'm not sure what you mean. What is the robot's goal? To follow the program detailed in the first paragraph?

Let's say Robot-1 genuinely has the goal to kill terrorists. If a hacker were to try to change its programming to "make automobiles" instead, Robot-1 would do anything it could to thwart the hacker; its goal is to kill terrorists, and letting a hacker change its goal would mean more terrorists get left alive. This sort of stability, in which the preference remains a preference regardless of context are characteristic of my definition of "goal".

This "blue-minimizing robot" won't display that kind of behavior. It doesn't thwart the person who places a color inversion lens on it (even though that thwarts its stated goal of "minimizing blue"), and it wouldn... (read more)

[-]Richard_Kennaway13y140

When you say the robot has a "different goal", I'm not sure what you mean. What is the robot's goal? To follow the program detailed in the first paragraph?

The robot's goal is not to follow its own program. The program is simply what the robot does. In the environment it is designed to operate in, what it does is destroy blue objects. In the vocabulary of control theory, the controlled variable is the number of blue objects, the reference value is zero, the difference between the two is the error, firing the laser is the action is takes when the error is positive, and the action has the effect of reducing the error. The goal, as with any control system, is to keep the error at zero. It does not have an additional goal of being the best destroyer of blue objects possible. Its designers might have that goal, but if so, that goal is in the designers, not in the system they have designed.

In an environment containing blue objects invulnerable to laser fire, the robot will fail to control the number of blue objects at zero. That does not make it not a control system, just a control system encountering disturbances it is unable to control. To ask whether it is still a control ... (read more)

[-]Scott Alexander13y120

Now I'm very confused. I understand that you think humans are PCT systems and that you have some justifications for that. But unlike humans, we know exactly what motivates this robot (the program in the first paragraph) and it doesn't contain a controlled variable corresponding to the number of blue objects, or anything else that sounds PCT.

So are you saying that any program can be modeled by PCT better than by looking at the program itself, or that although this particular robot isn't PCT, a hypothetical robot that was more reflective of real human behavior would be?

As for goals, if I understand your definition correctly, even a behaviorist system could be said to have goals (if you reinforce it every time it pulls the lever, then its new goal will be to pull a lever). If that's your definition, I agree that this robot has goals, and I would rephrase my thesis as being that those goals are not context-independent and reflective.

2Richard_Kennaway13y

I am saying that this particular robot (without the add-on human module) is a control system. It consists of nothing more than that single control system. It contains no representation of any part of itself. It does not reflect on its nature, or try to find other ways of achieving its goal. The hierarchical arrangement of control systems that HPCT (Hierarchical PCT) ascribes to humans and other living organisms, is more complex. Humans have goals that are instrumental towards other goals, and which are discarded as soon as they become ineffective for those higher-level goals. Behaviourism is a whole other can of worms. It models living organisms as stimulus-response systems, in which outputs are determined by perceptions. PCT is the opposite: perceptions are determined by outputs.

8Scott Alexander13y

I agree with you that behaviorism and PCT are different, which is why I don't understand why you're interpreting the robot as PCT and not behaviorist. From the program, it seems pretty clearly (STIMULUS: see blue -> RESPONSE: fire laser) to me. Do you have GChat or any kind of instant messenger? I feel like real-time discussion might be helpful here, because I'm still not getting it.

[-]Richard_Kennaway13y180

I agree with you that behaviorism and PCT are different, which is why I don't understand why you're interpreting the robot as PCT and not behaviorist. From the program, it seems pretty clearly (STIMULUS: see blue -> RESPONSE: fire laser) to me.

Well, your robot example was an intuition pump constructed so as to be as close as possible to stimulus-response nature. If you consider something only slightly more complicated the distinction may become clearer: a room thermostat. Physically ripped out of its context, you can see it as a stimulus-response device. Temperature at sensor goes above threshold --> close a switch, temperature falls below threshold --> open the switch. You can set the temperature of the sensor to anything you like, and observe the resulting behaviour of the switch. Pure S-R.

In context, though, the thermostat has the effect of keeping the room temperature constant. You can no longer set the temperature of the sensor to anything you like. Put a candle near it, and the temperature of the rest of the room will fall while the sensor remains at a constant temperature. Use a strong enough heat source or cold source, and you will be able to overwhelm the cont... (read more)

[-]Scott Alexander13y110

Okay, we agree that the simple robot described here is behaviorist and the thermostat is PCT. And I certainly see where you're coming from with the rats being PCT because hunger only works as a motivator if you're hungry. But I do have a few questions:

There are some things behaviorism can explain pretty well that I don't know how to model in PCT. For example, consider heroin addiction. An animal can go its whole life not wanting heroin until it's exposed to some. Then suddenly heroin becomes extraordinarily motivating and it will preferentially choose shots of heroin to food, water, or almost anything else. What is the PCT explanation of that?
I'm not entirely sure which correlation studies you're talking about here; most psych studies I read are done in an RCT type design and so use p-values rather than r-values; they can easily end up with p < .001 if they get a large sample and a good hypothesis. Some social psych studies work off of correlations (eg correlation between being observer-rated attractiveness and observer-rated competence at a skill); correlations are "lamentably low" in social psychology because high level processes (like opinion formation, social i

... (read more)

6erratio13y

Rats don't always choose drugs over everything else Summary: An experimenter thought drug addiction in rats might be linked to being kept in distressing conditions, made a Rat Park to test the idea, and found that the rats in the enriched Rat Park environment ignored the morphine on offer. EDIT: apparently the study had methodological issues and hasn't been replicated, making the results somewhat suspect, as pointed out by Yvain below

5Scott Alexander13y

I hate to admit I get science knowledge from Reddit, but the past few times this was posted there it was ripped apart by (people who claimed to be) professionals in the field - riddled with metholodogical errors, inconsistently replicated, et cetera. The fact that even its proponents admit the study was rejected by most journals doesn't speak well of it. I think it's very plausible that situation contributes to addiction; we know that people in terrible situations have higher discount rates than others and so tend to short-term thinking that promotes that kind of behavior, and certainly they have fewer reasons to try to live life as a non-addict. But I think the idea that morphine is no longer interesting and you can't become addicted when you live a stimulating life is wishful thinking.

1erratio13y

Damn. Oh well, noted and edited in to the original comment.

4Scott Alexander13y

Well, like I said, all I have to go on is stuff people said on Reddit and one failed replication study I was able to find somewhere by a grad student of the guy who did the original research. The original research is certainly interesting and relevant and does speak to the problems with a very reductionist model. This actually gets to the same problem I'm having looking up stuff on perceptual control theory, which is that I expect a controversial theory to be something where there are lots of passionate arguments on both sides, but on both PCT and Rat Park, when I've tried to look them up I get a bunch of passionate people arguing that they're great, and then a few scoffs from more mainstream people saying "That stuff? Nah." without explaining themselves. I don't know whether it's because of Evil Set-In-Their-Ways Mainstream refusing to acknowledge the new ideas, or whether they're just so completely missing the point that people think it's not worth their while to respond. It's a serious problem and I wish that "skeptics" would start addressing this kind of thing instead of debunking ghosts for the ten zillionth time.

2Richard_Kennaway13y

Just a brief note to say that I do intend to get back to this, but I've been largely offline since the end of last week, and will be very busy at least until the end of this month on things other than LessWrong. I would like to say a lot more about PCT here than I have in the past (here, here, and in various comments), but these things take me long periods of concentrated effort to write. BTW, one of the things I'm busy with is PCT itself, and I'll be in Boulder, Colorado for a PCT-related meeting 28-31 July, and staying on there for a few days. Anyone around there then?

1pjeby13y

The PCT learning model doesn't require reinforcement at the control level, as its model of memory is a mapping from reference levels to predicted levels of other variables. I.e., when the rat notices that the lever-pressing is paired with food, a link is made between two perceptual variables: the position of the lever, and the availability of food. This means that the rat can learn that food is available, even when it's not hungry. Where reinforcement is relevant to PCT is in the strength of the linkage and in the likelihood of its being recorded. If the rat is hungry, then the linkage is more salient, and more likely to be learned. Notice though, that again the animal's internal state is of primary importance, not the stimulus/response. In a sense, you could say that you can teach an animal that a stimulus and response are paired, but this isn't the same as making the animal behave. If we starved you and made you press a lever for your food, you might do it, or you might tell us to fork off. Yet, we don't claim that you haven't learned that pressing the lever leads to food in that case. (As Richard says, it's well established that you can torture living creatures until they accede to your demands, but it won't necessarily tell you much about how the creature normally works.) In any case, PCT allows for the possibility of learning without "reinforcement" in the behaviorist sense, unless you torture the definition of reinforcement to the point that anything is reinforcement. Regarding the leptin/ghrelin question, my understanding is that PCT as a psych-physical model primarily addresses those perceptual variables that are modeled by neural analog -- i.e., an analog level maintained in a neural delay loop. While Powers makes many references to other sorts of negative feedback loops in various organisms from cats to E. coli, the main thrust of his initial book deals with building up a model of what's going on, feedback-loopwise, in the nervous system and brain, no

-1MaoShan13y

To answer question 3, one could perform the experiment of surgically balancing leptin and ghrelin and not feeding or otherwise nourishing the subject. If the subject eventually dies of starvation, I would say the second theory is more likely.

3Perplexed13y

Outstanding comment - particularly the point at the end about the candle cooling the room. It might be worthwhile to produce a sequence of postings on the control systems perspective - particularly if you could use better-looking block diagrams as illustrations. :)

[-]JenniferRM13y100

My interpretation of this interaction (which is fascinating to read, btw, because both of you are eloquently defending a cogent and interesting theory as far as I can tell) is that you've indirectly proposed Robot-1 as the initial model of an agent (which is clearly not a full model of a person and fails to capture many features of humans) in the first of a series of articles. I think Richard is objecting to the connections he presumes that you will eventually draw between Robot-1 and actual humans, and you're getting confused because you're just trying to talk about the thing you actually said, not the eventual conclusions he expects you to draw from your example.

If he's expecting you to verbally zig when you're actually planning to zag and you don't notice that he's trying to head you off at a pass you're not even heading towards, its entirely reasonable for you to be confused by what he's saying. (And if some of the audience also thinks you're going to zig they'll also see the theory he's arguing against, and see that his arguments against "your predicted eventual conclusions" are valid, and upvote his criticism of something you haven't yet said. And both of you are... (read more)

5pjeby13y

I suspect Richard would say that the robot's goal is minimizing its perception of blue. That's the PCT perspective on the behavior of biological systems in such scenarios. However, I'm not sure this description actually applies to the robot, since the program was specified as "scan and shoot", not "notice when there's too much blue and get rid of it.". In observed biological systems, goals are typically expressed as perception-based negative feedback loops implemented in hardware, rather than purely rote programs OR high-level software algorithms. But without more details of the robot's design, it's hard to say whether it really meets the PCT criterion for goals. Of course, from a certain perspective, you could say at a high level that the robot's behavior is as if it had a goal of minimizing its perception of blue. But as your post points out, this idea is in the mind of the beholder, not in the robot. I would go further as to say that all such labeling of things as goals occurs in the minds of observers, regardless of how complex or simple the biological, mechanical, electronic, or other source of behavior is.

[-]JamesAndrix13y220

I suspect Richard would say that the robot's goal is minimizing its perception of blue. That's the PCT perspective on the behavior of biological systems in such scenarios.

This 'minimization' goal would require a brain that is powerful enough to believe that lasers destroy or discolor what they hit.

If this post were read by blue aliens that thrive on laser energy, they'd wonder they we were so confused as to the purpose of a automatic baby feeder.

4pjeby13y

From the PCT perspective, the goal of an E. coli bacterium swimming away from toxins and towards food is to keep its perceptions within certain ranges; this doesn't require a brain of any sort at all. What requires a brain is for an outside observer to ascribe goals to a system. For example, we ascribe a thermostat's goal to be to keep the temperature in a certain range. This does not require that the thermostat itself be aware of this goal.

1MarkusRamikin13y

< If this post were read by blue aliens that thrive on laser energy, they'd wonder they we were so confused as to the purpose of a automatic baby feeder. Clever!

4Scott Alexander13y

Although I find PCT intriguing, all the examples of it I've found have been about simple motor tasks. I can take a guess at how you might use the Method of Levels to explain larger-level decisions like which candidate to vote for, or whether to take more heroin, but it seems hokey, I haven't seen any reputable studies conducted at this level (except one, which claimed to have found against it) and the theory seems philosophically opposed to conducting them (they claim that "statistical tests are of no use in the study of living control systems", which raises a red flag large enough to cover a small city) I've found behaviorism much more useful for modeling the things I want to model; I've read the PCT arguments against behaviorism and they seem ill-founded - for example, they note that animals sometimes auto-learn and behaviorist methodological insistence on external stimuli shouldn't allow that, but once we relax the methodological restrictions, this seems to be a case of surprise serving the same function as negative reinforcement, something which is so well understood that neuroscientists can even point to the exact neurons in charge of it. Richard's PCT-based definition of goal is very different from mine, and although it's easily applicable to things like controlling eye movements, it doesn't have the same properties as the philosophical definition of "goal", the one that's applicable when you're reading all the SIAI work about AI goals and goal-directed behavior and such. By my definition of goal, if the robot's goal were to minimize its perception of blue, it would shoot the laser exactly once - at its own visual apparatus - then remain immobile until turned off.

5pjeby13y

Ironically, quite a lot of human beings goals would be more easily met in such a way, and yet we still go around shooting our lasers at blue things, metaphorically speaking. Or, more to the point, systems need not efficiently work towards their goals' fulfillment. In any case, your comments just highlight yet again the fact that goals are in the eye of the beholder. The robot is what it is and does what it does, no matter what stories our brains make up to explain it. (We could then go on to say that our brains have a goal of ascribing goals to things that appear to be operating of their own accord, but this is just doing more of the same thing.)

0Richard_Kennaway13y

Can you spell out the philosophical definition? My previous comment, which I posted before reading this, made only a vague guess at the concept you had in mind: "this sort of conscious, reflective, adaptive attempt to achieve what we 'really' want".

2Scott Alexander13y

I think we agree, especially when you use the word "reflective". As opposed to, say, a reflex, which is an unconscious, nonreflective effort to acheive something which evolution or our designers decided to "want" for us. When the robot's reflection that shooting the hologram projector instead of the hologram fails to motivate it to do so, I start doubting its behaviors are goal-driven, and suspecting they're reflexive.

-4orthonormal13y

Every time you bring up PCT, I have to bring up my reasons for concluding that it's pseudoscience of the worst sort. (Note that this is an analysis of an experiment that PJ Eby himself picked to support his claims.)

-1pjeby13y

Actually, Yvain brought it up. Which linking I don't mind a bit, since you're effectively linking to my reply as well, which is then followed by your hasty departure from the thread with a claim that you'd answer my other points "later"... with no further comment for just under two years. Guess it's not "later" yet.. ;-) (Also, anyone who cares to read upthread from that link can see where I agreed with you about Marken's paper, or how much time it took me to get you to state your "true rejection" before you dropped out of the discussion. AFAICT, you were only having the discussion so you could find ammunition for a conclusion you'd reached long before that point.) You also seem to have the mistaken notion that I'm an idea partisan, i.e., that because I say an idea has some merit or that it isn't completely worthless, that this means I'm an official spokesperson for that idea as well, and therefore am an Evil Outsider to be attacked. Well, I'm not, and you're being rude. Not only to me, but to everyone in the thread who's now had to listen to both your petty hit-and-run pa(troll)ing, and to me replying. So, I'm out of here (the subthread), but I won't be coming back later to address any missed points, since the burden is still on you to actually address any of the many, MANY questions I asked you in that two-year-old thread, for which you still have yet to offer any reply, AFAICT. I entered that discussion with a willingness to change my mind, but from the evidence at hand, it seems you did not. (Note: if you do wish to have an intelligent discussion on the topic, you may reach me via the old thread. I'm pre-committing not to reply to you in this one, where you can indulge your obvious desire to score points off an audience, vs. actually discussing anything.)

3orthonormal13y

Thanks for the poisoned well, but I don't intend to abuse the last word. I think more highly of you now than I did when we had our prior altercation, but it remains true that I've seen zero experimental evidence for PCT in a cognitive context, and that Marken's paper is an absolute mathematical sham. There may be valid aspects to PCT, but it hasn't yet justified its use as a cognitive theory, and I feel that it's important to note this whenever it comes up on Less Wrong. (Incidentally, the reason I trailed off in that thread is because I'd done something that in retrospect was poor form: I'd written up a full critique of the Marken paper before I asked you whether you thought it constituted experimental evidence, and I was frustrated that you didn't walk into the trap. If we both agree that the paper is pseudoscience, though, there's nothing left to add.) P.S. I don't doubt that you've had success working with people through a PCT framework, but I suspect that it's a placebo effect: a sufficiently fuzzy framework gives you room to justify your (usually correct) unconscious intuitions about what's going on, and grants it the gravitas of a deep-sounding theory. (You might do just as well if you were a Freudian.) That's one reason why I discount anecdotal evidence of that form.

0SarahSrinivasan13y

I recall that a big problem we had before was trying to unpack what different people meant by the words "goal", "model", etc. But your description of at least this distinction you're drawing between the things which you're calling "goals" and the things which you're calling "programs" is very good, IMO!

[-]Eliezer Yudkowsky13y200

This robot is not a consequentialist - it doesn't have a model of the world which allows it to extrapolate (models of) outcomes that follow causally from its choices. It doesn't seem to steer the universe any particular place, across changes of context, because it explicitly doesn't contain a future-steering engine.

5Vladimir_Nesov13y

Heh, it's pretty much exactly what I said.

[-]DanielLC13y140

What exactly is meant by the robot having a human-level intelligence? Does it have two non-interacting programs: shoot blue and think?

5Will_Sawin13y

This seems to be the key point. Everything interesting about the whole project of human rationality is contained in the interaction between the parts of us that think and the parts of us that do. All of the theories Yvain is criticising are about, ultimately, explaining and modeling the relationship between these two entities

[-]Kaj_Sotala13y140

Ah, excellent. This post comes at a great time. A few weeks ago, I talked with someone who remarked that although decision theory speaks in terms of preferences and information being separate, trying to apply that into humans is fitting the data to the theory. He was of the opinion that humans don't really have preferences in the decision theoretic sense of the word. Pondering that claim, I came to the conclusion that he's right, and have started to increasingly suspect that CEV-like plans to figure out the "ultimate" preferences of people are somewhat misguided. Our preferences are probably hopelessly path-, situation- and information-dependent. Which is not to say that CEV would be entirely pointless - even if the vast majority of our "preferences" would never converge, there might be some that did. And of course, CEV would still be worth trying, just to make sure I'm not horribly mistaken on this.

The ease at which I accepted the claim "humans don't have preferences" makes me suspect that I've myself had a subconscious intuition to that effect for a long time, which was probably partially responsible for an unresolved disagreement between me and Vladimir Nesov earlier.

I'll be curious to hear what you have to say.

[-]XiXiDu13y190

...CEV-like plans to figure out the "ultimate" preferences of people are somewhat misguided. Our preferences are probably hopelessly path-, situation- and information-dependent.

This is off-topic but since you mentioned it and since I don't think it warrants a new post, here are my latest thoughts on CEV (a convergence of some of my recent comments originally posted as a response to a post by Michael Anissimov):

Consider the difference between a hunter-gatherer, who cares about his hunting success and to become the new clan chief, and a member of lesswrong who wants to determine if a “sufficiently large randomized Conway board could turn out to converge to a barren ‘all off’ state.”

The utility of the success in hunting down animals and proving abstract conjectures about cellular automata is largely determined by factors such as your education, culture and environmental circumstances. The same hunter gatherer who cared to kill a lot of animals, to get the best ladies in its clan, might have under different circumstances turned out to be a vegetarian mathematician solely caring about his understanding of the nature of reality. Both sets of values are to some extent mutua... (read more)

7ShardPhoenix13y

Is becoming the best hunter really one of the primitive man's terminal values? I would say his terminal values are more things like "achieving a feeling of happiness, contentment, and pride in one's self and one's relatives". The other things you mention are just effective instrumental goals.

3Kaj_Sotala13y

I mostly agree with this. I think that the idea of desires converging if “we knew more, thought faster, were more the people we wished we were, and had grown up closer together” relies on assumptions of relatively little self-modification. Once we get uploads and the capability for drastic self-modification, all kinds of people and subcultures will want to use it. Given the chance and enough time, we might out-speciate the beetle (to borrow Anders Sandberg's phrase), filling pretty much every corner of posthuman mindspace. There'll be minds so strange that we won't even recognize them as humans, and we'll hardly have convergent preferences with them. Of course, that's assuming that no AI or mind with a first-mover advantage simply takes over and outcompetes everyone else. Evolutionary pressures might prune the initial diversity a lot, too - if you're so alien that you can't even communicate with ordinary humans, you may have difficulties paying the rent for your server farm.

0Scott Alexander13y

At the end of this, I'm going to try to argue that something like CEV is still justified. Before I started thinking it through I was hoping that taking an eliminativist view of preferences to its conclusion would help tie up the loopholes in CEV, and so far it hasn't done that for me, but it hasn't made it any harder either.

2ewbrownv13y

CEV has worse problems that worries about convergence. The big one is that it's such a difficult thing to implement that any AI capable of doing so has already crossed the threshold of extremely dangerous transhuman capability, and there's no real solution to how to regulate its behavior while it's in the process of working on the extrapolation. It could very well turn the planet into computronium before it gets a satisfactory implementation, by which point it doesn't much matter what result it arrives at.

3Will_Sawin13y

Presumably it matters if it then turns the planet back?

7ewbrownv13y

Even if you're the type who thinks a Star Trek transporter is a transportation device rather than a murder+clone system, there's no reason to think the AI would have detailed enough records to re-create everyone. Collecting that level of information would be even harder than getting enough to extrapolate CEV. So I suppose it might matter to the humanity it re-creates, assuming it bothers. But we'd all still be dead, which is a decidedly suboptimal result.

2Will_Sawin13y

Well, a neverending utopia fit to the exact specifications of humanity's CEV is still pretty darn good, all things considered.

-2Will_Newsome13y

Related: I recommend to those who think that CEV is insufficiently meta that they read CFAI, and try to go increasingly meta from there instead. Expanding themes from CFAI to make them more timeless is also recommended; CFAI is inherently more timeless than CEV -- that's semi-personal jargon but perhaps the gist is sufficiently hinted at. Note that unlike metaness, timelessness is often just a difference of perspective or emphasis. I assert that CEV is a bastardized popularization of the more interesting themes originally presented in CFAI, and should not be taken very seriously. CFAI shouldn't either--most of it is useless--but it at least highlights some good intuitions. Edit: I do not mean to recommend proposing solutions or proposing not-solutions, I recommend the meta-level strategy of understanding and developing intuitions and perspectives.

[-]Perplexed13y100

... the mistake began as soon as we started calling it a "blue-minimizing robot".

Agreed. But what kind of mistake was that?

Is "This robot is a blue-minimizer" a false statement? I think not. I would classify it as more like the unfortunate selection of the wrong Kuhnian paradigm for explaining the robot's behavior. A pragmatic mistake. A mistake which does not bode well for discovering the truth, but not a mistake which involves starting from objectively false beliefs.

[-]endoself13y90

Why does the human-level intelligence component of the robot care about blue? It seems to me that it is mistaken in doing so. If my motor cortex was replaced by this robot's program, I would not conclude that I had suddenly started to only care about blue, I would conclude that I had lost control of my motor cortex. I don't see how it makes any difference that the robot always had it actions controlled by the blue-minimizing program. If I were the robot then, upon being informed about my design, I would conclude that I did not really care about blue. My human-level intelligence is the part that is me and therefore contains my preferences, not my motor cortex.

[-]Scott Alexander13y230

If my motor cortex was replaced by this robot's program, I would not conclude that I had suddenly started to only care about blue, I would conclude that I had lost control of my motor cortex.

I predict this would not happen the way you anticipate, at least for some ways to cash out 'taking control of your motor cortex'. For example, when a neurosurgeon uses a probe to stimulate a part of the motor cortex responsible for moving the arm, and eir patient's arm moves, and the neurosurgeon asks the patient why ey moved eir arm, the patient often replies something like "I had an itch", "it was uncomfortable in that position", or "What, I'm not allowed to move my arm now without getting grilled on it?"

Or in certain forms of motor cortex damage in which patients can't move their arm, they explain it by saying "I could move my arm right now, I just don't feel like it" or "That's not even my real arm, how could you expect me to move that?".

Although I won't get there in a while, part of my thesis for this sequence is that we infer our opinions from our behaviors, although it's probably more accurate to say that our behaviors feed back to the same processes that generate our opinions and can alter them. If this is true, then there are probably very subtle ways of taking control of your motor cortex that would leave your speech centers making justifications for whatever you did.

[-]Vladimir_Nesov13y150

I'd be very surprised if this worked on me for more than, say, a day. Even if the intuition that I'm the one in control doesn't go away, I expect to eventually notice that it's actually false and consciously choose to not take it into account, at least in verbal reasoning. Has it been tried (on someone more qualified than a random patient)? If it doesn't work, the effect should be seen as rather more horrible than just overriding one's limb movement.

1MaoShan13y

Interesting. It's been two days since your cortex has been swapped out, and you seem not to have noticed. Affect slight annoyance at what you will believe to be my weak attempt at humor. This will enable you to produce a consistent, rational narrative which will satisfy your pattern-recognition wetware. That will use less glucose, making you more energy-efficient. People who care about the environment care about energy efficiency, you know. You do care about the environment, don't you? Of course you do. So, let's get to it now...

0Mqrius12y

I've got pretty bad akrasia. I want to do things, but then I do other things. Intuitively, I feel like I'm me, and I'm in control. Rationally, not so much. Rationally speaking the answer to "Am I in control?" depends a lot on how broad you define "I". Is my rational mind in control? No way. Is my brain as a whole in control? Yeah, mostly. Do excuses automatically pop up when I avoid work? Definitely. "I wanted to relax." "I got distracted." "I hate working." Having some rationality allows me to see through them though, which I presume puts me in the "someone more qualified than a random patient" category you mention. I'm not sure if this is exactly what Yvain is referring to, I just want to shine a light on the matter from a different angle.

1matthew-m13y

I find this curious. When a physician taps my knee with a knee tapping hammer I don't think to myself "I chose to jerk my leg." I experience it as something out of my control. Perhaps endoself was mistaken in placing the robots programing in the motor complex, but I believe the point was that in the human experience there are two kinds of reactions: those we have at least some form of conscious control over and those we have no conscious control over; and the robot's blue minimizing programing would fall into the later. Thus the robot would not experience the blue minimization as anything other than a strange reflex triggered by the color blue.

[-]Khaled13y60

Without overfitting, the robot has the goal of shooting at what it sees blue. It achieves its goal. What I get from the article is that the human intelligence mis interprets the goal. Here I see the definition of a goal to equal what the program is written to do, hence it seems inevitable that the robot wll achieve its goal (if there is a bug in the code that misses shooting a blue object every 10 days, then this should be considered part of the goal as well, since we are forced to define the goal in hindsight, if we have to define one)

-2MugaSofer12y

Do you reason similarly for humans?

4Richard_Kennaway12y

It is almost proverbial that intentions are better revealed by deeds than by words.

-3MugaSofer12y

You're right. Man, I can't believe I've been wasting my time persuading people to sign up for cryonics, because if they wanted to live, they would have already done so! I can't believe I didn't realize this before.

0Khaled12y

I can't think of another way to reason - does our brain dictate our goal, or receives a goal from somewhere and makes an effort to execute it accurately? I'd go with the first option, which to me means that whatever our brain (code) is built to do is our goal. The complication in the case of humans might be the fact that we have more than one competing goal. It is as if this robot has a multi-tasking operating system, with one process trying to kill blue objects and another trying to build a pyramid out of plastic bottles. Normally they can co-exist somehow with some switching between processes or by just one process "not caring" about doing some activity at the current instance. It gets ugly when the robot finds a few blue bottles. Then the robot becomes "irrational" with one process destroying what the other is trying to do. This is simply when you are on a healthy diet and see a slice of chocolate cake - you're processes are doing their jobs, but they are competing on resources - who gets to move your arms? Let's then imagine that we have in our brains a controlling (operating) system that can get to decide which process to kill when they are in conflict. Will this operating system have a right and wrong decision? Or will whatever it does be the right thing according to its code - or else it wouldn't have done it?

-2MugaSofer12y

I was thinking of, y'know, biased people. Also known as me and everyone I've ever met. Telling them "don't worry, whatever you're already doing is what you really want" does not seem Friendly.

[-]Psy-Kosh13y60

Does that "human level intelligence module" have any ability to actually control the robot's actions, or just to passively observe and then ask "why did I do that?" What're the rules of the game, as such, here?

4Scott Alexander13y

I don't think it's saying anything too shocking to admit this is all a metaphor for people; I'm going to be pushing the view that people's thoughts and words are a byproduct of the processes that determine behavior rather than directly controlling them. I anticipate providing at least a partial answer to your question in about two weeks; if that doesn't satisfy you, let me know and we can talk about it then.

[-]Richard_Kennaway13y120

this is all a metaphor for people

One that presents consciousness as an epiphenomenon. In the version of the robot that has human intelligence, you describe it as bolted on, experiencing the robot's actions but having no causal influence on them, an impotent spectator.

Are your projected postings going to justify this hypothesis?

4Scott Alexander13y

I hope so. Let's see.

2asr13y

My first thought was that this was pointing towards an epiphenomenal view of consciousness. But I think it's actually something more radical and more testable. Yvain, check me if I get this wrong, but I think you're saying that "our conscious verbal acts -- both internally and externally directed -- do not primarily cause our actions." Here is an experiment to test this: have people perform some verbal act repeatedly, and see if it shifts their actions. This happens to be a well known motivational and behavior-alteration technique, beloved of football teams, political campaigns, governments, and religions, among other organizations. My impression is that it works to a point, but not consistently. Has anybody done a test of how catechisms, chants, and the like shape behavior?

0jsalvatier13y

I hope that he explicitly deals with this. By the way, I didn't know the actual definition of epiphenomenon, which is " is a secondary phenomenon that occurs alongside or in parallel to a primary phenomenon".

4Richard_Kennaway13y

But then again... See also.

2Perplexed13y

I take this to be an elliptical way of suggesting that Yvain is offering a false dichotomy in suggesting a choice between the notion of thoughts being in control of the processes determining behavior and the notion of thoughts being a byproduct of those processes. I agree. Thoughts are at one with (are a subset of) the processes that determine behavior.

2Scott Alexander13y

I'm not so sure. Using the analogy of a computer program, we could think of thoughts either as like the lines of code in the program (in which case they're at one with, or in control of, the processes generating behavior, depending on how you want to look at it) or you could think of thoughts as like the status messages that print "Reticulating splines" or "50% complete" to the screen, in which case they're byproducts of those processes (very specific, unnatural byproducts, to boot). My view is closer to the latter; they're a way of allowing the brain to make inferences about its own behavior and to communicate those inferences. Opaque processes decide to go to Subway tonight because they've heard it's low calorie, then they produce the verbal sentence "I should go to Subway tonight because it's low calorie", and then when your friend asks you why you went to Subway, you say "Because it's low calorie"). The tendency of thoughts to appear in a conversational phrasing ("I think I'll go to Subway tonight") rather than something like "Dear Broca's Area - Please be informed that we are going to Subway tonight, and adjust your verbal behavior accordingly - yours sincerely, the prefrontal cortex" is a byproduct of their use in conversation, not their internal function. Right now I'm just asserting that this is a possibility and that it's distinct from thoughts being part of the decision-making structure. I'll try to give some evidence for it later.

5MixedNuts13y

If you make the old mistake of confusing thoughts in general with analytic, reflective, verbal, serial internal monologue, I'm going to be sad. I find this rather alien. Some processes are opaque, but that kind definitely isn't. Something (hunger, time, memory of previously made plans, whatever) triggers a reusable pick-a-sandwich-shop process; names and logos of nearby stores come up; associated emotions and concepts come up; weights associated to each shift - an image of those annoying health freaks who diet all the time upvotes "tasty" and downvotes "low calorie"; eventually they stabilize, create an image of myself going to Subway rather than somewhere else, and hand it over to motor control. If something gets stuck at any point, the process stops, a little alarm rings, and internal monologue turns to it to make it come unstuck. If not, there are no verbal thoughts at any point.

2Scott Alexander13y

Probably time to start being sad; I'm mostly going to use "thoughts" that way. But I think what I'm talking about holds for any definition of "thought" where it's a mental activity accessible to the conscious mind. I recognize different people use internal monologue to a different degree than others, but whether you decide with a monologue or with vague images of concepts, I think the core idea that these are attempts to turn subjective processes into objects for thought, usually so that you can weave a social narrative around them, remains true.

1Perplexed13y

You may have missed a subtlety in my comment. In your grandparent, you said "people's thoughts and words are a byproduct ...". In my comment, I suggested "Thoughts are at one with ...". I didn't mention words. If we are going to focus on words rather than thoughts, then I am more willing to accept your model. Spoken words are indeed behaviors - behaviors that purport to be accurate reports of thoughts, but probably are not. Perhaps we should taboo "thought", since we may not be intending the word to designate the same phenomenon.

2Psy-Kosh13y

What I meant was that if the intelligence part is utterly passive re the behavior, then I'm unsure as to how strong a metaphor it is for human behavior. Yes, we sometimes don't know why we do things, or we have inaccurate models of ourselves as far as our explanations of why we do what we do. But the "intelligence part" having absolutely no affect on the actions? But, perhaps all will be resolved in two weeks. :)

[-]zxcvbnm13y50

Robin Hanson would also argue that the robot wants to minimize things that are 'far', since blue is 'far'

[-]Johnicholas13y50

One of the obvious extensions of this thought experiment is to posit a laser-powered blue goo that absorbs laser energy, and uses it to grow larger.

This thought experiment also reminds me: Omohundro's arguments regarding likely uFAI behavior are based on the AI having goals of some sort - that is, something we would recognize as goals. It's entirely possible that we wouldn't perceive it as having goals at all, merely behavior.

4lessdazed13y

Please share your thoughts more often, that wasn't obvious to me at all. I not unoften say things I think are obvious and get surprised by extreme positive or negative reactions to them.

-1timtyler13y

If is isn't goal directed, it isn't intelligent. Intelligence is goal-directed - by most definitions.

4scav13y

To the extent that intelligence solves problems, then yes, problem-solving modules of an intelligent entity have at least short-term goals. But whether the entity itself has goals is a different question. I can imagine a sufficiently dangerous uFAI having no consistent goals we would recognise, and all of its problem-solving behaviour, however powerful and adaptable, being at a level we wouldn't call intelligence (e.g. the uFAI could include a strategic module like a super-powerful but clearly non-sentient chess computer, which would not in itself have any awareness or intent, just problem-solving behaviour.) Actually, I'm not going to disagree with you about definitions of intelligence. But I suspect most of them are place-holders until we understand enough to dissolve the question "what is intelligence?" a bit.

0[anonymous]10y

Be suspicious of arguments from definition. Why must intelligence be goal-directed? Why is this an integral part of the definition of intelligence, if it must be?

[-]AdeleneDawner13y50

Is there a sense in which we can conclude that the robot is a blue-minimizing robot, in which we can't also conclude that it's an object-minimizing robot that happens to be optimized for situations where most of the objects are blue or most of the backgrounds are non-blue? (Perhaps it's one of a set of robots, or perhaps the ability to change its filter is an intentional feature.)

[-]DCrowe13y40

A couple of points here. First, as other people seem to have indicated, there does seem to be a problem with saying 'the robot has human level intelligence/self-reflective insight' and simultaneously that it carries out its programming with regards to firing lasers at percepts which appear blue unreflectively, in so far as it would seem that the former would entail that the latter would /not/ be done unreflectively. What you have here are two seperate and largely unintegrated cognitive systems one of which ascribes functional-intentional properties to thin... (read more)

[-]Khaled13y40

Shouldn't the human intelligence part be considered part of the source code? With its own goals/value functions? Otherwise it will be just a human watchig the robot kind of thing.

[-]Kaj_Sotala13y30

I wonder how many people upvoted this post less for the ideas expressed and more because they like robots.

I think I upvoted it for the ideas, but can't honestly guarantee that the "oooh, shiny cool robot analogy appeals to my geeky heart" factor wouldn't have made me upvote even if I found the ideas uninteresting.

[-]Will_Newsome13y190

I wonder how many people upvoted this post less for the ideas expressed and more because they like robots.

I wonder how many people upvoted this post less for the ideas expressed and more because they like Yvain.

[-]Jonathan_Graehl13y100

I like how Yvain creates clarity. This takes a lot of effort. I'd like to encourage his effort.

4Nisan13y

I upvote articles and comments that promise future articles that I want to read.

0MatthewBaker13y

ditto :) I never have enough reading material.

0Bagricula13y

I up-voted because: 1. Writing-style gives me a strong internal impression of clarity / comprehension. I enjoy this sensation and think it correlates with understanding, and so am trying to promote more of it. 2. Blue feels very soothing and I think contributes to the sensation of clarity / comprehension. I up-vote soothing sensation. 3. I like the possible direction this could take in terms of microeconomic utility, revealed preferences, and so on for understanding human intelligence. So my up-vote is payment for expected future ideas.

[-]lukeprog13y30

I haven't thought much about why it feels like we have goals (desires), so I look forward to that! I do think it's quite possible that eliminativism about 'beliefs' and 'desires' will turn out to be the best way to go. Certainly, the language of 'reinforcers' fits well with our understanding of the reward-learning system in the brain.

2Nisan13y

Chapter 2 of Gary Drescher's Good and Real attempts to address why it feels like we have goals. If the robot could learn to destroy hologram projectors and adapt to color-inverting lenses, the assertion "the robot wants to minimize blue objects" would have some explanatory virtue.

4lukeprog13y

Yes, 'folk psychology' (beliefs+desires=intentional action) is a compelling theory because it works so successfully in everyday social interactions, but I'm wondering if Yvain has something more planned for why it feels (inside) like we have goals. My guess is that it's because we use the same folk theory to explain our own behavior as to explain others' behavior, but I'm wondering if Yvain has something else in mind.

2Scott Alexander13y

Nope, pretty much that. I'll be presenting a few studies to justify it, but I'm sure you've seen them before.

0lukeprog13y

Ah. Well. They are fun studies. :)

1MichaelVassar13y

It's not obvious that folk psychology works well at all except regarding motor actions.

[-]Duncan13y20

I consider all of the behaviors you describe as basically transform functions. In fact, I consider any decision maker a type of transform function where you have input data that is run through a transform function (such as a behavior-executor, utility-maximizer, weighted goal system, a human mind, etc.) and output data is generated (and in the case of humans sent to our muscles, organs, etc.). The reason I mention this is that trying to describe a human's transform function (i.e., what people normally call their mind) as mostly a behavior-executor or jus... (read more)

0BrandonReinhart13y

Is "transform function" a technical term from some discipline I'm unfamiliar with? I interpret your use of that phrase as "operation on some input that results in corresponding output." I'm having trouble finding meaning in your post that isn't redefinition.

[-][anonymous]13y20

Well-written and insightful. This post reminds me very much of the free will sequence, except that the reduction is on the level of genes and the adaptations they code for rather than physical laws. I look forward to seeing the rest of the sequence, and I'm interested to see how you will dissolve the feeling that our behavior and actions are goal-oriented.

[-]TruePath13y10

The entire example is deeply misleading. We model the robot as a fairly stupid blue minimizer because this seems to be a good succinct description of the robots entire externally observable behavior and would cease to do so if it also had a speaker or display window with which it communicated it's internal reflections.

So to retain the intuitive appeal of describing the robot as a blue minimizer the robots human level intelligence must be walled off inside the robot unable to effectively signal to the outside world. But so long as the human level intellig... (read more)

[-]Volndeau13y10

EDIT: It just clicked after finishing my thought.

If its human handlers (or itself) want to interpret that as goal directed behavior, well, that's their problem.

I was thrown off by all the comments about the robot and its behavior. This is more about the comparison of behavior-executor vs. utility-maximizer, not the robot. EDIT:

Perhaps I am missing the final direction of this conversation, but I think the intelligence involved in the example has mapped the terrain and failing to update the map once it has been seen to not be correct.

Watching the robot

... (read more)

[-]XiXiDu13y10

The human-level intelligence version of the robot will notice its vision has been inverted. It will know it is shooting yellow objects. It will know it is failing at its original goal of blue-minimization.

I don't see that, why would it "care" if its goal isn't complex enough to allow it to care about the subversion of its sensors? I mean, the level of intelligence seems irrelevant here. Intelligence isn't even instrumental to such simple goals because all it "wants" is to fire a laser at blue objects. Its utility function says nothing about maximizing its efficiency or anything like that.

1torekp13y

Unfortunately, your question is unanswerable, at least until Yvain invents the rest of the story. We haven't been told what goals, if any, are embodied in the intelligent part of the code. Strike that "if any" part, though - I think we can infer that it has goals from the specification that it has human-level intelligence. And even infer something about what some of the goals are like (truth-seeking, for example). We also haven't been told the relationship between blue-zapping code and intelligence - whether it is physically possible, for example, for the intelligent processes to modify the blue-zapping code modules. Edit: Psy-Kosh raised similar questions.

[-]RPMcMurphy10y00

Take three common "broad" or "generally-categorizable" demographics of minds: Autistic people, Empaths (lots of mirror neurons dedicated to modeling the behavior of others), Sociopaths, or "Professional Psychopaths" (high-functioning without mirror neurons, responsible for most systemic destruction, precisely because they can appear to be "highly functional and productive well-respected citizens"), Psychopaths (low-functioning without mirror neurons, most commonly an "obvious problem"). All of the prior hum... (read more)

[-][anonymous]10y00

This article got me thinking about a few things. Centrally, there is no necessary condition on things which really do have goal directed behaviour to maximise. Revealed preference theory in economics effectively identifies the choices people make with their preference. Thus humans are seen as maximisers of their preferences in an almost trivial sense, if preferences are also assumed to fall under a ranking (another thing economists assume). Yet this goal orientated explanation in terms of rationality could be wrong for two reasons: firstly, the choices cou... (read more)

[-]AuroraDragon11y00

Also, the assumption that the laser is for destruction may be false. What is the laser tags or scans blue objects? Maybe for a space probe, or a scan on enemy robots? What if the robot makes the assumption that the laser is for destruction when it's not, or vise-versa? Does it really matter, when it is programmed to do it anyway? What if the AI is destroying, or thinks it's destroying fellow robots? What if it does not want to? Maybe this is an example of religion from assumption? Because it is 'destroying' blue things, it assumes that is it's purpose. Beh... (read more)

[-]Said Achmiz11y00

the robot continuously analyzes the average RGB value of the pixels in the camera image; if the blue component passes a certain threshold, the robot stops, fires its laser at the part of the world corresponding to the blue area

I note that if the the robot only looks at the blue RGB component, then it will end up firing its laser not just at blue things (low R, low G, high B), but also white things (high R, high G, high B), fuchsia things (high R, low G, high B), teal things (low R, high G, high B), and variants of said colors. "Blue-minimizing" is not even a correct description!

[-]MugaSofer12y00

The robot is a behavior-executor, not a utility-maximizer.

Or it only gets disutility from seeing blue objects :P

[-]Curiouskid13y00

It seems like we need to taboo the word "goal" and replace it with several different things. 1. Actions. 2. Conscious (verbal) intentions (past intentions may conflict with present intentions). 3. "unconscious intentions" (this should probably be tabooed, but I haven't figured out how best to do it. Perhaps an unconscious intention is something which we do, but we don't know why or we confabulate why.).

it seems that Anna Salamon's and Hanson's interpretations could simply be viewed as changing verbal intentions and the question if how to gain goal (intention) stability.

[-][anonymous]13y00

A lot of this is discussed at length in the book The Ecological Approach to Visual Perception by Gibson. He put out a pretty radical idea that all cognition is goal driven, and perception is the action of converting sensory energy into scored models (scores are computed as anticipated experience of goal-fulfillment). Fundamentally, all thinking is planning. There's quite a bit of literature coming out on this in the fields of online learning theory and actionable information theory, in particular by Soatto at UCLA.

[-]B_Rightmore11y-10

Curious that you worded it: "...we would be tempted to call this robot a blue-minimizer." Then you said the robot "wants". The entire discussion which followed is invalid, because everyone involved assumed human characteristics to an electronic mechanical device. Robots do not have "goals" either, nor self-motivation. IT has no concept of reducing blue objects. Your "temptation" was your sensed instinct that you were stepping over the line of limitations. Even the origin of the word "robot" meaning "forced labor" is inaccurate.

[-]PhilGoetz13y-10

And are you a utility-optimizer, or a behavior-executor?

[-]Pavitra13y-10

Yes, yes, yes!

[-]XiXiDu13y-10

I think all of these explanations hold part of the puzzle, but that the most fundamental explanation is that the mistake began as soon as we started calling it a "blue-minimizing robot".

Your post reminded me of a quote by Eliezer Yudkowsky:

An electron is not a billiard ball, and it’s not a crest and trough moving through a pool of water. An electron is a mathematically different sort of entity, all the time and under all circumstances, and it has to be accepted on its own terms.
The universe is not wavering between using particles and waves,

... (read more)

0lessdazed13y

Even what a human is is not stable. My skin regenerates...my teeth are mineralized...I have some fillings...I eat some grapes...and spit out the pits...pull on some pants...and go drive a car. How am I qualitatively non-arbitrarily not my car? The Voyager Horcrux has naked humans, but if hermit crabs were intelligent, would they send the probe up with an image of hermit crabs without shells? I am not sure.

[-]Decius12y-20

What is the difference between a smart 'shoot lasers at "blue" things' robot and a really dumb 'minimize blue' robot with a laser?

2Andreas_Giger12y

A really smart 'shoot lasers at "blue" things' robot will shoot at blue things if there are any, and will move in a programmed way if there aren't. All its actions are triggered by the situation it is in; and if you want to make it smarter by giving it an ability to better distinguish actually-blue from blue-looking things, then any such activity must be triggered as well. If you program it to shoot at projectors that project blue things it won't become smarter, it will just shoot at some non-blue things. If you paint it blue and put a mirror in front of it it will shoot at itself, and if you program it to not shoot at blue things that look like itself it won't become smarter, it will just shoot at fewer blue things. If anything it shoots at doesn't cease to be blue or you give it a blue laser or camera lens, it will just continue shooting because it doesn't care about blue things or shooting; it just shoots when it sees blue. It certainly won't create blue things to shoot at. A really dumb 'minimize blue' robot with a laser will shoot at anything blue it sees, but if shooting at something doesn't make it stop being blue, it will stop shooting at it. If there's nothing blue around it will search for blue things. If you paint it blue and put a mirror in front of it it will shoot at itself. If you give it a blue camera lens it will shoot at something, stop shooting, shoot at something different, stop shooting, move around, shoot at something, stop shooting, etc, and eventually stop moving and shooting altogether and weep. If instead of the camera lens you give it a blue laser it will become terribly confused.

-2Decius12y

Why would a dumb 'minimize blue' robot search for blue things? Current condition: I see no blue things. Expected result of searching for blue things: There are more blue things. Expected result of not searching for blue things: There are no blue things. Where 'blue' is defined as 'eliciting a defined response from the camera'. Nothing outside the view of the camera is blue by that definition.

3TheOtherDave12y

...and a sufficiently non-dumb 'minimize blue' robot using that definition would disconnect its own camera.

-2Decius12y

Right. If we want anthropomorphic behavior, we need to have multiple motivations.

0wedrifid12y

A "dumb robot" is assumed to be a weak optimizer. It isn't defined to be actively biased and defective in known ways related to the human experience 'denial'. Sure you can come up with specific ways that an optimizer could be broken and draw conclusions about what that particular defective robot will do. But these don't lead to a conclusion where it makes sense to rhetorically imply the inconceivability of a robot that doesn't have that particular bug.

-2Decius12y

I'm putting 'dumb' at roughly the level of cognition of a human infant, lacking object permanence. Human toddler intelligence counts as 'smart'. I'm considering a strict reward system: negatrons added proportional to the number of blue objects detected.

0wedrifid12y

Then, as per the grandparent, the answer to the rhetorical question "Why would a dumb 'minimize blue' robot search for blue things?" is because it doesn't happen to be a robot designed with the exact same peculiarities and weaknesses of a human infant.

-2Decius12y

Lack of object permanence isn't a peculiar weakness. The ability to spontaneously leave Plato's cave is one of the things that I reserve for 'smart' actors as opposed to 'dumb' ones.

0Qiaochu_Yuan12y

A smart robot that shoots lasers at blue things will shoot at blue things it models as being there even if it can't see them.

0wedrifid12y

A smart 'shoot lasers at "blue" things' robot may title the cosmic commons with blue things to be shot (exactly what it tiles the cosmic commons with depends on the details of implementation.) A really dumb minimize blue robot... um... it'll shoot blue things. Probably. And sometimes miss.

0Decius12y

I didn't mean a smart "maximize blue things shot with lasers" robot. Although I suppose creating blue things to shoot is a reasonable action to take once all the easily accessible blue things have been destroyed. Oddly enough, a similar behavior has been noted in AA and other rehab support groups; when there are no more easily accessible addicts to cure, someone will relapse. That's perfectly rational behavior for a group that wants to rehabilitate people, even if it isn't conscious.

[+]VManuel13y-60

[+][comment deleted]4y20

Deleted by Abhimanyu Pallavi Sudhir, 10/28/2024

Moderation Log

161Comments