Context for LW audience: Ramana, Steve and John regularly talk about stuff in the general cluster of agency, abstraction, optimization, compression, purpose, representation, etc. We decided to write down some of our discussion and post it here. This is a snapshot of us figuring stuff out together.

Hooks from Ramana:

• Where does normativity come from?
• Two senses of "why" (from Dennett): How come? vs What for? (The latter is more sophisticated, and less resilient. Does it supervene on the former?)
• An optimisation process is something that produces/selects things according to some criterion. The products of an optimisation process will have some properties related to the optimisation criterion, depending on how good the process is at finding optimal products.
  • The products of an optimisation process may or may not themselves be optimisers (i.e. be a thing that runs an optimisation process itself), or may have goals themselves. But neither of these are necessary.
  • Things get interesting when some optimisation process (with a particular criterion) is producing products that are optimisers or have goals. Then we can start looking at what the relationship is between the goals of the products, or the optimisation criteria of the products, vs the optimisation criterion of the process that produced them.
• If you’re modeling “having mental content” as having a Bayesian network, at some point I think you’ll run into the question of where the (random) variables come from. I worry that the real-life process of developing mental content mixes up creating variables with updating beliefs a lot more than the Bayesian network model lets on.

A central question regarding normativity for me is “Who/what is doing the enforcing?”, “What kind of work goes into enforcing?” 

Also to clarify, by normativity I was trying to get at the relationship between some content and the thing it represents. Like, there’s a sense of the content is “supposed to” track or be like the thing it represents. There’s a normative standard on the content. It can be wrong, it can be corrected, etc. It can’t just be. If it were just being, which is how things presumably start out, it wouldn’t be representing.

Intrinsic Purpose vs Purpose Grounded in Evolution

Steve

As you know, I totally agree that mental content is normative - this was a hard lesson for philosophers to swallow, or at least the ones that tried to “naturalize” mental content (make it a physical fact) by turning to causal correlations. Causal correlations was a natural place to start, but the problem with it is that intuitively mental content can misrepresent - my brain can represent Santa Claus even though (sorry) it can’t have any causal relation with Santa. (I don’t mean my brain can represent ideas or concepts or stories or pictures of Santa - I mean it can represent Santa.)

Ramana

Misrepresentation implies normativity, yep.

In the spirit of recovering a naturalisation project, my question is: whence normativity? How does it come about? How did it evolve?

How do you get some proto-normativity out of a purely causal picture that’s close to being contentful?

Steve

So one standard story here about mental representation is teleosemantics, that roughly something in my brain can represent something in the world by having the function to track that thing. It may be a “fact of nature” that the heart is supposed to pump blood, even though in fact hearts can fail to pump blood. This is already contentious, that it’s a fact the heart is supposed to pump blood - but if so, it may similarly be a fact of nature that some brain state is supposed to track something in the world, even when it fails to. So teleology introduces the possibility of misrepresentation, but it also brings suspicious normativity. It involves a “supposed to”, and the question becomes whether there really is a fact of nature about what something is supposed to do, and what about nature could ground such facts.

Ramana

I agree that looking at functionality (reasons - i.e. the “What for?” version of “why”) is a related - and perhaps simpler - thing to normativity. So maybe the simpler question is: how does “how come?” become “what for?” (Dennett explored this a bunch in From Bacteria to Bach and Back.)

Steve

So here as you know the Dennettian (and indeed standard teleosemantic) answer is natural selection - that in fact there is this optimizer, as you say, that brought about other optimizers. We want to say that the frog’s ability to snap its tongue out in the right direction toward a fly when the frog gets the right retinal stimulation is an evolved process that is supposed to catch flies, even though it might mistakenly catch other flying debris sometimes. The story there is that the mechanism in the frog is best explained by its inheriting functional apparatus that was shaped by natural selection, and that best explanation points to what the apparatus is for.

Ramana

I guess I’m more interested in what makes something an optimiser than in the fact that evolution produced them. Like mechanically what makes it the case that something is an optimiser. Or mechanically what makes it the case that something has a function. Is there a mechanical (aka causal) answer to that kind of question? Is something else needed?

It does not seem good enough to answer “evolution” to the question “why does this thing have function X?” – I don’t want to know how it came to be a thing that has that function, I want to know what in itself makes it have that function.

If “explanation” is an ingredient in the answer, I feel like that’s interesting, because it means it’s not a totally causal story.

John

It does not seem good enough to answer “evolution” to the question “why does this thing have function X?”

Example (which may or may not emphasize what Ramana has in mind): sometimes evolution spits out random stuff that isn’t functional, like e.g. junk DNA sequence. And therefore there must be more to function than just having evolved?

Ramana

Yeah that’s right. I think swamp hearts are more directly examples. They still pump. What about them makes them have that function? (Or might someone mistakenly claim that although they behave like pumps they don’t really have the function since their evolutionary story is missing. I think that’s a mistake since it misses the point of why I cared about things having functions in the first place.)

Steve

I’m glad you brought up swamp-cases. To the extent this is for an audience: swampman cases are ones where, as a thermodynamic miracle, a creature molecularly identical to Ramana happens to form in a swamp. We have the strong intuition that swamp-Ramana’s heart is supposed to pump blood, even though the action of swamp-Ramana’s heart is not best explained by evolutionary inheritance. Similarly we might think that when swamp-Ramana asks for the nearest coffee shop - as he would, given he has all the same causal mechanisms moving his vocal chords as regular Ramana - he is really thinking about coffee.

This is at least a bit odd, though, since swamp-Ramana has never encountered coffee. Perhaps a more central example: when swamp-Ramana says “this swamp smells terrible”, he’s really thinking about the swamp around him, and its smell. But standard teleosemantics would have to deny this. And in fact many philosophers do bite the bullet here, and say that swamp-hearts do not have the function to pump blood, and swamp-Ramana does not have mental content. But I agree this is not a great bullet to bite.

John

swampman cases are ones where, as a thermodynamic miracle, a creature molecularly identical to Ramana happens to form in a swamp.

Context for audience: yes, Ramana specifically, he’s been used as a prototypical example in the philosophical literature for decades. Ramana’s parents attempted to raise him to perfectly match the prototypical swampman example as well as possible, and successfully started a minor religion off the gambit.

Steve

Smarty-pants.

Ramana

Like I really get the feeling that evolution is a red herring in trying to answer the original question about whence normativity (or whence functionality). Evolution is definitely an interesting answer, but it’s answering a slightly different question.

Steve

Yes. There are other proposals for the natural grounds of teleology, but they all have their problems. I have come to think maybe Terence Deacon has made good progress on this question, as may shortly come up.

This bit is somewhat idiosyncratic to me: one thing that I think can be clarifying here is distinguishing when something has a function “for itself”. For example, a hammer clearly has a function to pound nails - but this is somewhat obviously in virtue of others’ intentions. For something to have a function “for itself” means that the system somehow benefits from the performance of that function. (Actually now that I think of it, something like this is one of the going alternatives / complements to teleology-from-evolution, but I can’t credit the philosophers / papers offhand.) I tend to think of this in terms of “derived teleology” (like the hammer) and “original teleology” (like a living system).

Ramana

Curious about the other proposals, kinda. Esp. the best one (Deacon’s) according to you. Also keen to get some agreement on our stance on evolution here.

I think the hammer has a function to pound nails for itself.

It also has the function to pound nails because it was designed that way by something with an intention.

I want to know about the hammer’s function for itself nonetheless. What about it gives it that? Or maybe would you disagree that it has such a function for itself?

My “for itself” is a bit different from yours maybe. I don’t mean that it needs it for survival or something. I mean more like there is a fact about this arrangement of matter that makes it functional, and I want to understand what that fact is. I’m very curious about whether it is a fact about matter, or if I need to bring in something else like an observer, a designer, an explainer, etc.

Steve

Yeah so many have the intuition - I think including me - that the hammer’s function is only relative to intentions. Suppose you start using the hammer only as a doorstop instead. Did it gain a new function? Does it still have the function to pound nails? I think most philosophers want to say about these kinds of cases that “eh, there’s no deep fact here, it depends on our intentions.” But I think most philosophers do not want to say such things about hearts, eyes, etc.

Dennett I think is different from most philosophers in this regard - if I read him right, he doesn’t think there are deep facts about teleology anywhere, and there is no “original teleology”. The paper I keep going back to on this is his “Evolution, error, and intentionality” (also reprinted in his book The Intentional Stance). This makes it the case that there aren’t deep facts about mental content, either. He is happy enough to buy this.

John

I think the hammer has a function to pound nails for itself.

Possibly clearer example here: consider a jigsaw puzzle. I would guess that a whole variety of alien mind-types would look at jigsaw puzzle pieces and go “man, it really seems like these are ‘supposed to’ fit together in this particular way”.

So we could say that “the jigsaw puzzle pieces’ function is to fit together”, in the intentional stance sense (i.e. lots of minds converge to view them that way). But then the hot question is: what’s the real pattern about puzzle pieces which those alien minds pick up on, and view as “the jigsaw puzzle pieces’ function is to fit together”?

Ramana

(I would say the hammer always had the doorstop function.)

Ramana (later)

Summarising one of my thoughts here:

Claim: There is a way of being that is “having the function of pumping blood” or “having the function of pounding nails” or “having goals” etc.

Swamp things are this way, as are their natural counterparts. Being these ways is in some sense (thermodynamically?) very unlikely, surprising, or miraculous. I see “evolution” (and derived intentionality more proximally in some cases) as being the answer to the following question: How did such a miraculous way of being nevertheless come into existence?

That question, and answer, – although fascinating and worthy of study in its own right – is a distraction from the question I really care about (and which I think is more pertinent to alignment). My question is something more like this: What is it about this way of being that makes it “having the function of X”? How does it work?

(Note: this is subtly different from the question “How does a heart work?”. It’s a tiny bit more meta. “How does having a function work?”)

Steve (later)

I think I see what you mean about the hammer. My notion of a function “for itself” is supposed to be that the functional mechanism somehow benefits the thing of which it’s a part. (Of course hammers can benefit carpenters, but we don’t tend to think of the hammer as a part of the carpenter, only a tool the carpenter uses. But I must confess that where that line is I don’t know, given complications like the “extended mind” hypothesis.) I think what you’re suggesting, Ramana, is a somewhat different plausible thesis: that a hammer intrinsically has the function to pound nails. It sounds like you want to say, look, it’s a fact about this hammer that it would be really helpful pounding nails, and not so helpful for trying to stay afloat on water, and fairly helpful as a doorstop, etc.

I also like John’s example of the jigsaw puzzle. I think this too gets at the idea that look, it’s just a fact about this physical system that it has some natural state. I am not sure, but we might be distracted in the jigsaw puzzle case by our knowledge that it’s an artifact (that is, someone designed it with the pieces’ fitting together in mind). Suppose, John, that I spent decades going around the world finding sizable pieces of driftwood that could be assembled into a sphere with almost no empty space. Would you say those pieces were supposed to fit together, as a fact about those pieces? In fact of course they do fit together - but supposed to? Maybe I’m too soaked in theory, but I don’t have such an intuition that jigsaw pieces are supposed to fit together in any “intrinsic” sense - only in the “derived” sense that it is part of our shared intentions to make and solve puzzles like that.

This reminds me of Ramana’s question about what “enforces” normativity. The question immediately brought me back to a Peter Railton introductory lecture I saw (though I may be misremembering / misunderstanding / misquoting, it was a long time ago). He was saying that real normativity is not like the old Windows solitaire game, where if you try to move a card on top of another card illegally it will just prevent you, snapping the card back to where it was before. Systems like that plausibly have no normativity to them, when you have to follow the rules. In a way the whole point of normativity is that it is not enforced; if it were, it wouldn’t be normative.

So Ramana I think you’re asking: what makes it the case that something really has a function? And I think the Dennett-style answer here is: first, forget really having a function, there’s no such thing. But certainly some things are more aptly described as having this or that function. When the first proto-wings were in transition from balancers or seed-gatherers (or whatever they used to do) to glide-enhancers, there was no deep fact about when they first became glide enhancers. But you will make more predictive headway treating the wings as evidence for the fact that the creature might glide or gather seeds, rather than as evidence for the fact that the creature needs to fit in small spaces or something.

Meanwhile it seems like in the nature of normativity that we can’t give a purely causal answer to when something has a function. It has to be a different kind of answer. Maybe information theory won’t do either - this is one place (of many) where I’m confused. Maybe it’s a dilemma: if the notion of information already has some normativity built into it, it can be the basis of teleology, but not of course eliminate basic normativity. And if we can give a purely non-normative account of information, it might not be sufficient to ground normativity.

I am tempted to try to work out Deacon’s proposal for basic teleology next, though I’m not sure how productive that would be.

Intrinsic Purpose from Minimum Description

John

Suppose, John, that I spent decades going around the world finding sizable pieces of driftwood that could be assembled into a sphere with almost no empty space. Would you say those pieces were supposed to fit together, as a fact about those pieces?

I would say that the pieces acquire the purpose of fitting together with almost no empty space via the act of you gathering them. Scattered randomly amongst all the other driftwood of the world, the pieces would lack that purpose. But throw them in one pile, and the pile has the function of fitting together with almost no empty space.

(I admit I have a background model which generated that answer; I didn’t just intuit it.)

Steve

Yeah - that seems like the right answer, but it gives up on the “intrinsic” purpose of the pieces fitting together, and the same seems to apply to the jigsaw puzzle. They only are “supposed” to fit together in virtue of others’ intentions.

John

I don’t think that answer gives up on intrinsic purpose, it just assigns intrinsic purpose to the pile (or the whole puzzle) rather than individual pieces.

Steve

Well I’m not sure if this is fair intuition-pushing, but consider these cases:

• Swamp-pile: the driftwood that fits together perfectly just happens, by weird chance, to wash up to shore in a loose pile. Are they supposed to fit together? 
• Near-piles: the pieces are spread a bit further apart. Now a bit further. How close do they have to be before they are supposed to fit together?

John

For the swamp-pile, any intrinsic answer would say they’re supposed to fit together. At the very least, we can hopefully agree that the swamp pile is well-modeled as having been generated by someone optimizing for the pieces to fit together (for purposes of intrinsic questions, not necessarily history), and therefore will have whatever real patterns people typically recognize as hallmarks of a-thing-optimized-for-some-“purpose”.

That answer also points to how we should handle the near-piles: when the wood is in one pile, it is many many orders of magnitude more likely that such a pile would arise from someone optimizing the pile of wood to fit together, than that it would arise by chance. As the sub-piles spread, that statistical advantage falls, until at some point the sub-piles do resemble random chance.

Steve

Okay I like this response; as you know I like “compressibility” standards. And it’s true the driftwood pile I stipulate is suspiciously compressible, and gets more so as they are bunched into their sphere, and less so as the pile gets spread apart and the pieces’ locations are more random.

But I think this standard is for a thing (a “composition” of smaller parts), and not necessarily for an optimized thing. Like: a rock can be isolated from its environment by something like its internal compressibility, the real pattern that’s there. But this does not imply the rock was optimized for something or has a purpose / function.

But maybe you can say more about how compressibility would also imply function? There may be something there …

John

Sure, here’s the compression version: sometimes, the shortest program to generate some data will look like

Argmax_x objective(x) subject to constraints(x)

If the data can be generated by a program of that form, using much fewer bits than a program of some other form, then the data “is well-modeled as having been optimized for <objective>”. That’s basically what we’re invoking in the driftwood case: the pile of wood which fits together perfectly can be (nontrivial claim!) generated from a program which involves an argmax, using much fewer bits than it would take without an argmax.

And as the pile is scattered further and further, the number of extra bits of compression achievable by using an argmax falls.

Steve

Yeah I like this direction and I want to explore it, I hadn’t thought in just that way before, compressing via an optimization. A hesitation though: I can also compress a rock much further if I know the initial conditions and the laws of physics that led to its formation. I mean, everything is (plausibly) best compressed by “initial conditions of the universe + laws of nature”.

But yes maybe there’s a good intentional-stance route here - like sure, if we’re Laplacean demons the rock and the driftwood (and real agents) are all predictable from physics, but given our mere mortality we can do better with compression using an optimization algorithm.

John

A couple subtleties:

• Often what we should really think about is compressibility given some background data already available to us. So for instance, we may have already noticed that all of physics can be compactly represented as an argmax over negative action (i.e. “principle of least action”), but that still leaves a bunch of stuff unknown (e.g. initial conditions) which still leaves us with a bunch of uncertainty. So we probably want to think about how another argmax (beyond the usual least-action) can further compress our data.
• Notably, the “conditional on background data” part means that we probably want the objective and/or constraints to have short specifications conditional on background data/knowledge about the world, not necessarily unconditionally short specifications.
• We should maybe be thinking about quantilization rather than pure minimization/maximization, since in practice things-which-have-been-optimized usually haven’t been optimized “all the way”.

Steve

The first bullet leaves us with stance-dependent function though right? In a way that’s less objective than “intentional stance”, maybe (real pattern there for any heuristic shortcut to see)?

I’m not sure I follow bullet two. I sort of get bullet three but would want to see details.

Like can we come up with a toy model of a system that’s best compressed as being the result of an optimization process?

John

System which is (very likely) best compressed as being the result of an optimization process: weights (or full specification?) of a neural net which has been trained to play Go.

Steve

Huh, okay, nice. So in other words, instead of giving you all the raw weights, I can send a message which is something like: n layers of p nodes trained up on this loss, and you’re already most of the way there to recovering my weights, is that the idea?

John

… well, in practice, the weights will still look quite random after conditioning on that, but the residual will be properly random (i.e. incompressible), assuming we initialized with actually-random bits (which admittedly we usually don’t, we use a pseudo-random generator, but that’s not especially central to the point here).

Steve

Right, I have to add a noise term to my two-part message but it still plausibly compresses. I like this and will have to think about it more; it certainly seems to be in the spirit of the Intentional Stance.

Objectivity vs Subjectivity

Ramana

Is there an “intentional stance with teeth” in this? It seems possibly too general… – Let me expand on what I mean by “too general”: I feel like there’s something interesting/important to do with abstraction and concept-formation that is subjective that gets lost with naive/initial attempts to apply information theory to formalising things like goal/function/purpose/ontology via compressibility or things along those lines. I think I am worried that this argmax approach is gonna not have room for the real ambiguity (and simultaneously the real non-ambiguity) of the function of hammers we were discussing earlier. “Too general” was probably not the right complaint. It’s more like: too objective, or insensitive to important features.

Possible directions that come to mind (not exhaustive) for me: (a) this is not actually a problem because there will be a bunch of programs, with different ontologies or suggestions about how to interpret the situation, that are all roughly the same size, so the real ambiguity will remain in how to pick between them. (b) what I want will show up in the choice of universal machine/language (I’m skeptical about this being sufficiently detailed though). 

Steve

In response to Ramana: I guess I’m inclined toward the “more objective the better”, and that there’s plenty of subjectivity in the choice of universal Turing machine for doing the pattern abstraction. But John and I both want to hear more here.

The Dennett-ish thing to say would be: there may be multiple different optimizations that compress to different standards for different purposes. And that’s why there’s no one deep fact about the purpose of a thing. (Ramana: that’s my direction (a) above, arrived at independently.)

John

There is one other potential source of subjectivity (i.e. divergence in how different agents model things) here: different background data/knowledge on which they’re conditioning. Insofar as the compression of the system of interest is just one part of a bigger world-model, the key question is how much marginal compression is achieved by modeling the system as optimized-for-<objective> within that bigger world-model. And that means that e.g. the objective/constraints need only have short descriptions given whatever other stuff already has short names within the world-model.

So insofar as different agents have different background data/knowledge, and therefore different overall world-models, they could view different things as well-compressed using argmax/argmin/quantilization.

Ramana

Conditional compressibility has always been this obvious thing to do in the background of the information-theoretic approaches. But I think we’re now seeing how it could actually be a way to solve a bunch of issues. Mind-independent mind-dependent ontologies.

I particularly like not having to push all my desire for subjectivity into the choice of UTM. That never felt right to me. What does seem correct is having an objective measure of what patterns are real given a subjective/arbitrary starting point (an existing mind, community/culture, evolutionary history, whatever - the thing that provides the stuff to condition on)

Steve

Right, we’re looking for a roughly “mind-independent” (or if you like “model-independent”) sense in which some thing has a function, and we can still say that it is in fact compressible (given UTM choice) by this or that optimization procedure, whether some agent models it that way or not. So it’s not as objective as God’s-eye-view deep fact, but not as subjective as “I think it has this function (given my background knowledge etc), so it does.”

I am wondering now if we are capturing the normativity / teleology that I think we wanted in the very beginning. Like it’s one thing to say: the heart resulted from this optimization process, and it’s even well-modeled as having resulted from that process in swamp-Ramana. But it might be another thing to say, that heart should (is supposed to / has the function to) pump blood. But now I’ve lost track.

Ramana

I think we can get “swamp heart has the function of pumping blood” out of this, yeah. Swamp heart works for having the function of pumping blood, and “things that work for things we care about are supposed to continue to work and be resilient [under changes we care about them being resilient to]” could be a thing that uses the conditional/background info to turn “works” into normative conditions. 

John

(Minor quibble: I’d be careful about using “should” here, as in “the heart should pump blood”, because “should” is often used in a moral sense. For instance, the COVID-19 spike protein presumably has some function involving sneaking into cells, it “should” do that in the teleological sense, but in the moral sense COVID-19 “should” just die out. I think that ambiguity makes a sentence like “but it might be another thing to say, that the heart should pump blood” sound deeper/more substantive than it is, in this context.

So e.g. Ramana says “swamp heart has the function of pumping blood”, and I think that more directly says the thing we’re trying to talk about here without connotations of moral questions which we’re not trying to talk about here.) (Ramana: I agree with all this)

Steve

Yeah John on the one hand we want the should - that’s the normative part - but it’s a “pro tanto” should or something, like the nice COVID case. Maybe “supposed to” is sufficiently normative while being ethics-neutral.

So I think the main thing of interest to me is whether we’re recapturing the kind of possibility of misrepresentation that seems to imply something normative in the background, and that might be of interest when doing real interpretability work. Something like: “yes it was optimized for this but it’s not always or very good at it.”

Ramana I am not sure I follow the “things that work for things we care about …” Maybe we can do more on it next time.