LESSWRONG
LW

Comment Permalink
Ishaan70

I realize this has little to do with the actual point, but --

I think it does.

Like you, I think "savory" is a term for a certain type of non-sugar related goodness, of which the modern umami are a major component but not the only component. Things clearly get more savory when you salt to taste, for example, despite not hitting glutamate receptors more (I think). Yet pure salt isn't that savory.

It's not an empty category, it corresponds to a perceptual reality which arises for complicated reasons. Not all non-sweet things are savory. Water, bitterness, etc isn't sweet or savory, for example. Extreme sweetness prevents things from being savory, in the same way being burning-hot prevents a touch from being ticklish, but that doesn't mean that the absence of sweetness is truly the definition.

And while "empty categories" is a novel and interesting lens for me to look at things, I still think all the examples listed here suffer from the same flaws, for example:

Spirit - Calculus is neither physical nor spiritual. Spiritual specifically corresponds to meaningful things, and is tied into the subjection perception of a fundamental difference between inanimate and animate objects. The word "spiritual" pins down how some stuff is purposeful and meaningful and elevated.

The reductionist worldview makes it obvious how the intuition of spirits do not carve reality at the joints (much like how the understanding of taste receptors show the intuition of "savory" as not carving reality at the joints!) but that doesn't make them empty categories. Spirit does actually refer to a real thing that we perceive...it's just not a perception that easily withstands reductionist rigor. ...or you can try and say that spirits are made out of matter, thereby slightly altering the original meaning...that's what the "umami = savory = glutamate receptors" schema is doing.

Suppose we had a word that described an optical illusion, but we didn't know it was an optical illusion. Later, we discover that it is an optical illusion, and the reality isn't like we thought. Don't you think our original word describing our naive perceptions continues to mean something?

Originally, we had an illusion that agents and objects were fundamentally different and made words that described that intuition. Are the old words empty, just because reality turned out not to be that way?

The same argument applies to the other examples. Terminal values, for example. It feels like there are some things I value extrinsically (money) and others I value intrinsically (family). If after subsequent analysis it turns out I can't conveniently carve the complex reality of a tangle of neurons with that intuition, if I have to discard that intuition to really get how it works, does that really make the original idea empty?

As for why these are often defined in negative terms...it's often easier to communicate a definition by shooting down nearby concepts. You know from context that "savory" means pleasurable taste, and if you remove sweet from the flavor map, most of what remains falls under savory.

That meant if she found something she called a spirit, and I found that it had a mind containing information about our world, that would prove there were statistical regularities in its interaction with our world, and a configuration of parts to store the information, and it would by definition no longer be a spirit.

I'm not so sure about this. I bet if, when we first opened up the human skull, there was nothing there, and our motor neurons just mysteriously lit up, scientifically-minded people would still be talking seriously about souls and spirits to this day. As reductionists, we could hypothesize that there must be a spirit-mind, governed by spirit-laws, somewhere in a lawful spirit-world which intersects with our world at the spinal chord, and we'd do behavioral psych experiments to determine how it works and made reductionist models, I bet no dualists would feel uncomfortable with that explanation. It's just that reality turned out not to fundamentally separate agents and objects in that way. The only reason dualists and reductionists don't often get along is because reductionist scientific methods discovered the universe is monist, and that fact boggles our poor human intuitions the same way physics sometimes does.

I still think that the "empty category" concept might be useful. It would be nice to have an indicator of which intuitions were beginning to warp thinking and not adequately carve the reality as it revealed itself, and it's possible that the degree to which we have to resort to defining a concept by what it is not is an indicator that we're clinging to something that the evidence is pushing against.

TL:DR This unfortunately is getting too rambling to be useful because I'm thinking while I'm typing. I think what I'm grasping at is, while these so-called "empty" categories might be useful for identifying things which are simultaneously pleasing to the intuition and wrong, they aren't "not even wrong". They do in fact contain meaningful (but either incorrect or non-rigorous) content

...although (sorry, I know this was supposed to be a TL:DR) I guess they might become "not even wrong" if they're foisted into models that don't have room for them. The free will debate made much more sense back when it was more "Can a person really control their destiny, or is that foolish hubris and the gods decide all?" and I think at some deeper emotional level many people who try to debate undissolved free will are really asking "Am I free or are the physics gods bossing me around?"...but then the ground shifted and the debate was inappropriately transferred forced into an arena where the gods were replaced by a lawful universe and now it's all complete nonsense unless we do that "redefine savory as umami" trick and say that free will is implemented by laws, that's right, the "God" was you all along! (I call it a "semantic trick" because it solves problems via a redefinition of word meaning, but respectfully: it's a semantic trick with a long, illustrious history spanning from the Upanishads to Lesswrong's own Yudkowski and it really works for unconfusing people. When a redefined meaning seems to carve reality better than the original, it's more correct in some respects.)

The original question about free will was a valid question, the original constructs of spirits and so-on not empty, even if attempts to engage then analytically often generate nonsense until you understood what exactly that fuzzy human intuition was attempting to communicate.

Reply
Richard_Kennaway80

Suppose we had a word that described an optical illusion, but we didn't know it was an optical illusion. Later, we discover that it is an optical illusion, and the reality isn't like we thought. Don't you think our original word describing our naive perceptions continues to mean something?

For example, "sunrise".

Reply
See in context
by PhilGoetz
9 min read
68

22

Savory

What does "savory" mean when talking about food? Merriam-Webster says:

  • having a pleasant taste or smell
  • having a spicy or salty quality without being sweet
  • pleasing to the sense of taste especially by reason of effective seasoning
  • pungently flavorful without sweetness

Macmillan says:

  • a small piece of food that tastes of salt or spices and is not sweet

But when found in the wild, "savory" is usually contrasted with sweet, and is either freed from the "salt or spices" requirement, or used in a context that already implies "salty, spicy, or sweet." As this debate on chowhounds shows, plenty of cooks think "savory" means "not sweet." It is then not a category, but an uncategory, defined by what it is not.

Spirit as uncategory

Recently I had a conversation with a woman who wanted to know whether I really believed only in material things. I said, no; I also believed in magnetic fields and gravity, for instance.

She said those didn't count, because magnetic fields and gravity fields are explained by particle interactions. I said, maybe they are, but I still believed in them after I stopped believing in the soul, but before anybody told me they were mediated by particles.

She said gravity and magnetism are deterministic. I said I also believed in quantum mechanics. She said that didn't count as belief in something non-material either, because the things we see quantum-mechanical effects in aren't intelligent.

I was confused: What did being intelligent have to do with being material? She said that non-material things were spirits, and spirits were intelligent.

I should have immediately realized where this was going, but I pressed on. I said I was willing to believe that one could use gravitational forces to build a (very very large) computer, with a quantum-mechanical random number generator, and then implement an intelligent program on it, and would that count as a spirit?

By this time she was getting a little upset with me, and said, approximately, that a spirit is an intelligent being, not composed of analyzable simpler parts, whose interactions with our world are not subject to any physical laws or statistical regularities, yet which can perceive our actions and cause things to happen in our world.

That meant if she found something she called a spirit, and I found that it had a mind containing information about our world, that would prove there were statistical regularities in its interaction with our world, and a configuration of parts to store the information, and it would by definition no longer be a spirit.

She'd begun with the postulate that intelligence was unexplainable, perhaps because that was what she needed God for. So she defined a category that meant "intelligent beings that have no parts and no information and cannot be observed in our universe" and used it to explain human intelligence.

This category combined two important categories of uncategories: impossible uncategories defined not to be anything that exists, and inaccessible uncategories defined not to be anything that can be observed. Both are empty categories, defined so that you can't find any members.

There turn out to be a lot of these! Platonic forms, magic, soul, essence...

God as the ultimate empty category

Suppose you want your tribe to obey a set of rules. So you tell them the rules were made by an old man called God.

Pretty soon the tribe is tired of these rules, and they ask where God is, so they can argue with him. Well, you can't say he's three valleys over, or they might go there and look. You have to say he isn't anywhere they can go. God is an inaccessible uncategory.

Try this experiment on a religious friend: Tell him you think you might believe in God. Then ask him to list the qualities that define God. Argue with anything using the word "perfect", like "a perfect being, perfectly just, perfectly loving," etc. (Anything that requires perfection is an uncategory, but they aren't helpful just now.)

If you can get him to settle for a list of sufficient conditions like "created the universe," "can know anything in the universe," "has power over everything in the universe," etc., then tell him that, yes, you now believe in God. After he's finished congratulating you, explain that you have decided that it's almost certain, based on your priors, that we live in a simulation, and the being who runs this simulation is God. You call Him Fred. Most likely He's the super(universe) equivalent of a grad student.

Your religious friend may say that isn't believing in God. Go over the list of God's attributes he just made, and ask which one Fred doesn't have.

(Pro tip: Avoid trying this experiment on family members.)

Fred seems too real. Many difficulties with any given monotheistic religion have been pushed into the God concept and dismissed by adding another impossible quality to God. You could call God the ultimate uncategory, the node in a religious GUT where nonsense accretes. A God who exists might not have all those qualities. So the concept of God has been constructed and connotated to make sure anybody who exists can't be Him.

People seldom start religions by saying they're God. They say they're God's messenger, or maybe God's son. But not God. Then God would be this guy you saw stub his toe, and he'd end up like that guy in "The Man Who Would Be King."2

Empty categories protect false beliefs

Rarely, such uncategories may be created deliberately, such as Russell's "the set of all sets that do not contain themselves." But there's no need to suppose most were made deliberately.

Most new theories are bad theories. A bad theory whose parts are all open to inspection can be easily dismissed. But a bad theory that postulates an entity or category that has a definition that sounds satisfiable, but is not, is harder to dismiss. All the difficulties in the theory can be pushed into the empty category, where they are safe from attack, since no one can find an instance of the class. If the postulated entity is an agent (spirits, God, a homunculus), any qualities needed by the theory (absolute goodness, infallibility, oracular computational power) can be attributed to it without fear of disproof. So theories with empty categories are selected for and accumulate in religion, psychology, philosophy, and ethics, where people have beliefs that (A) are stated in terms of vague, abstract concepts, and that (B) they care about deeply enough to tolerate the phrase "it is obvious that" in a proof.

Empty categories like spirit and God are easy to catch. But there are more subtle empty categories.

Free Will

To ask if a mental process is made under free will, you start by observing how its inputs determine its distribution of outcomes. If the distribution has a single value, you say that it's deterministic. If it's random, you say that randomness isn't free will. If it has a probability distribution, you say it is a mix of determinism and randomness. If you can't define its inputs or outputs, you can't call it a process and can't ask whether free will was involved.

The only way you could decide someone had free will was if you found a homunculus inside them making their choices. "Free will" is an uncategory that defends the belief that humans are special and not made out of parts.

(I could be wrong. Consciousness is still mysterious.)

Non-computational consciousness

See John Searle's Chinese room argument. Or don't. It defines consciousness as an impossible uncategory, arguing that anything that can be understood obviously can't be conscious.

Merit

Note I did not say "virtue." "Virtue" is a category; societies list the behaviors they want out of people. Even if the list just says "Men: Violent. Women: Chaste," those are behaviors that can be observed. When the Greeks, who believed in fate, spoke of virtue, they didn't get so worked up over the "merit" aspect of it. Great warriors had virtue, even if they merely inherited it by being the son of a god, and even if, like Achilles, they didn't want to act heroically.

Christians, who believed in free will and an eternal destination dependent on it, came to think more in terms of merit. A person forced to do something gains no merit from it; neither does a person who did it accidentally. Merit exists only where free will does.

Because we believe in free will, we try to make social rules and laws give people what they "merit" rather than maximize social utility. Instead of making sure to imprison the criminal who has a brain injury that makes him violent, we set him free, because it isn't his fault. Instead of giving scholarships to students who score high on tests, to invest in our future, we give them to students who score high relative to their demographically-predicted scores.

But if you investigate the "merit" of any actual human, you'll find an endless regress of circumstances which all of the blame or credit eventually accumulates to.

Human terminal values

Humans are genes' way of reproducing themselves. Human behavior implements a utility function all of whose terminal values are statements about gene allele frequencies. As humans were not even aware of allele frequencies until recently, anything a human thinks it values cannot be a terminal value of a human utility function. As the terminal values of different alleles are at odds with each other, nearly all the values of any given human's utility functions are competing in zero sum games with the values of other humans.

A human society's values are ultimately stated in terms of the gene alleles common in that society. These tend to be the values we think of as human values, because they often supervene on rationality and must be expressed explicitly. But, again, they benefit genes, not humans.

Most of the time, this is fine with us. We get hungry; we eat. We're cold; we put on a jacket. Our interests are largely aligned with those of our genes. But if for some reason you want to know what "human terminal values" are, and collect them into a set of non-contradictory values, ethics gets untenable, because your terminal values benefit alleles, not humans, and play zero-sum games, not games with benefits to trade or compromise.

There are two schools of thought current now in psychology regarding human values. One says that evolution encodes human values directly, and so we're stuck today with values that evolved in the Stone Age. People from this school of thought often say that racism, xenophobia, homophobia, and/or rape are human values, and attempts to mitigate them go against human nature. (For example, [3], [4], [5], [6].)

The other school of thought says that evolution encodes the computational ability to choose behaviors that implement human values in the particular environment [7]. Hence, men may have preferred women with narrow waists a hundred years ago, but (they argue) have changed in some modern societies since the sexual revolution in the 1970s to prefer women with wider waists. People subconsciously compute that they no longer want to be racist and xenophobic in an era in which trading with other cultures is more beneficial to our genes than killing them.

Neither school of thought would allow that many things we think of as human values really are. Consider the reactions of a mother and father on finding out their son is gay. They may be able to accept this and value their son's happiness above social pressure. But all current scientific theories of human behavior would say that parents are programmed to want their children not to be gay. Parents who tolerate homosexuality in a child have overwritten their terminal genetic values with instrumental human values, period.

Neither school would propose that human terminal values contain anything we, as modern humans, value. Either we are Stone Age machines programmed to hate, rape, and kill, or we are computers subconsciously optimizing our reproductive efficiency, that will switch from toleration, liberalism, and rationality to racism, conservatism, and religion as soon as conditions change.

There is a school of thought that has tried to develop a consistent set of human terminal values that could be programmed into a symbolic logic. They first assume that there exists a finite set of concepts with which you can represent the world in a context-insensitive manner, as is required for symbolic logic. Then they have looked for a consistent minimal set of terminal values behind existing expressed human values. They resolve conflicts between values by prioritizing the values in this set, and have successfully come up with a set of values that, as you might predict, in many ways optimizes the genetic fitness of people who observe them. They take the genes' side in most reproductive issues, including homosexuality, incest, birth control, and abortion. We call this school of thought the Catholic Church.

ADDED: If you don't believe that humans, human preferences, and your feelings, were produced by evolution, this is not the place to have that argument.

2. If they do claim to have some sort of personal godhood, or not to be dictating the thoughts of some higher being, they'll speak in aphorisms and parables, or invent their own vocabularies, so that you can't argue with them (Jesus, Buddha, Hegel).

3. Thornhill & Palmer (2000). A Natural History of Rape. Cambridge, MA: MIT Press.

4. Santos, Meyer-Lindenberg, Deruelle (2010). Absence of racial, but not gender, stereotyping in Williams syndrome children. Current Biology 20(7):R307-308.

5. Edward Wilson (1975). Sociobiology. Cambridge, MA: Harvard University Press.

6. Steven Pinker (2002). The Blank Slate. NYC: Viking.

7. David Buller (2006). Adapting Minds: Evolutionary Psychology and the Persistent Quest for Human Nature. Cambridge, MA: MIT Press.

Personal Blog

22

Uncategories and empty categories
New Comment
68 comments, sorted by
top scoring
Click to highlight new comments since:
[-]tavaton100

I'm confused.

Why would human terminal values necessarily have to encompass or be compatible with genetic terminal values? Doesn't this assume whatever mental algorithm determines which instrumental values are most conducive to reproductive efficiency in any environment has access to explicit and accurate information about the genetic endgoal? If it's using heuristic proxies instead, and of course it is, since it's not possible to directly observe reproductive fitness, the proxies would then be human terminal values.

Reply
[-]PhilGoetz20

The genetic values are terminal because they don't serve other values and hence don't change. Human preferences serve genetic terminal purposes, so human preferences change with the environment. Also, they're heuristics more than values, so they're likely to be inconsistent and to sometimes counter terminal values.

Reply
[-]tavaton40

Precisely. So human values, terminal or not, are divorced from genetic terminal values. We may be survival machines, but that doesn't mean we're aware of it or think like we're aware of it, so that fact is only related to what our actual values are. It doesn't dictate it.

Reply
[-][anonymous]00

Why would human terminal values necessarily have to encompass or be compatible with genetic terminal values?

Because sometimes giving oneself existential angst is more important than having reflectively coherent definitions for concepts.

If it's using heuristic proxies instead, and of course it is, since it's not possible to directly observe reproductive fitness, the proxies would then be human terminal values.

Well that's the first-level notion rather than the considered-on-reflection-and-full-information version we're actually aiming at, but yes. But just go along for the ride and pretend to be scared when Azathoth jumps out at you.

Don't you know darkness and terror are just more philosophical than mere knowledge?

Reply
[-]PhilGoetz20

Don't you know darkness and terror are just more philosophical than mere knowledge?

So is sarcasm, apparently.

If you've got something to say, say it. If you have a "reflective equilibrium" approach that will allow you to find a consistent set of human values without going back to terminal values, cite a reference.

Reply
[-]Sniffnoy70

Huh. I realize this has little to do with the actual point, but -- I had generally taken "savory" to refer to the basic "umami" taste. The chowhounds discussion is in that respect a little... scary? Not sure that's the right word. That said, I'm not sure that (aside from the question asker, who may get the wrong idea from the provided answers) that the answerers actually think it means "not sweet"; they might have the same extension as me in mind, and are just failing to express it in a helpful way.

Reply
[-]Ishaan70

I realize this has little to do with the actual point, but --

I think it does.

Like you, I think "savory" is a term for a certain type of non-sugar related goodness, of which the modern umami are a major component but not the only component. Things clearly get more savory when you salt to taste, for example, despite not hitting glutamate receptors more (I think). Yet pure salt isn't that savory.

It's not an empty category, it corresponds to a perceptual reality which arises for complicated reasons. Not all non-sweet things are savory. Water, bitterness, etc isn't sweet or savory, for example. Extreme sweetness prevents things from being savory, in the same way being burning-hot prevents a touch from being ticklish, but that doesn't mean that the absence of sweetness is truly the definition.

And while "empty categories" is a novel and interesting lens for me to look at things, I still think all the examples listed here suffer from the same flaws, for example:

Spirit - Calculus is neither physical nor spiritual. Spiritual specifically corresponds to meaningful things, and is tied into the subjection perception of a fundamental difference between inanimate and animate objects. The word "spiritual" pins down how some stuff is purposeful and meaningful and elevated.

The reductionist worldview makes it obvious how the intuition of spirits do not carve reality at the joints (much like how the understanding of taste receptors show the intuition of "savory" as not carving reality at the joints!) but that doesn't make them empty categories. Spirit does actually refer to a real thing that we perceive...it's just not a perception that easily withstands reductionist rigor. ...or you can try and say that spirits are made out of matter, thereby slightly altering the original meaning...that's what the "umami = savory = glutamate receptors" schema is doing.

Suppose we had a word that described an optical illusion, but we didn't know it was an optical illusion. Later, we discover that it is an optical illusion, and the reality isn't like we thought. Don't you think our original word describing our naive perceptions continues to mean something?

Originally, we had an illusion that agents and objects were fundamentally different and made words that described that intuition. Are the old words empty, just because reality turned out not to be that way?

The same argument applies to the other examples. Terminal values, for example. It feels like there are some things I value extrinsically (money) and others I value intrinsically (family). If after subsequent analysis it turns out I can't conveniently carve the complex reality of a tangle of neurons with that intuition, if I have to discard that intuition to really get how it works, does that really make the original idea empty?

As for why these are often defined in negative terms...it's often easier to communicate a definition by shooting down nearby concepts. You know from context that "savory" means pleasurable taste, and if you remove sweet from the flavor map, most of what remains falls under savory.

That meant if she found something she called a spirit, and I found that it had a mind containing information about our world, that would prove there were statistical regularities in its interaction with our world, and a configuration of parts to store the information, and it would by definition no longer be a spirit.

I'm not so sure about this. I bet if, when we first opened up the human skull, there was nothing there, and our motor neurons just mysteriously lit up, scientifically-minded people would still be talking seriously about souls and spirits to this day. As reductionists, we could hypothesize that there must be a spirit-mind, governed by spirit-laws, somewhere in a lawful spirit-world which intersects with our world at the spinal chord, and we'd do behavioral psych experiments to determine how it works and made reductionist models, I bet no dualists would feel uncomfortable with that explanation. It's just that reality turned out not to fundamentally separate agents and objects in that way. The only reason dualists and reductionists don't often get along is because reductionist scientific methods discovered the universe is monist, and that fact boggles our poor human intuitions the same way physics sometimes does.

I still think that the "empty category" concept might be useful. It would be nice to have an indicator of which intuitions were beginning to warp thinking and not adequately carve the reality as it revealed itself, and it's possible that the degree to which we have to resort to defining a concept by what it is not is an indicator that we're clinging to something that the evidence is pushing against.

TL:DR This unfortunately is getting too rambling to be useful because I'm thinking while I'm typing. I think what I'm grasping at is, while these so-called "empty" categories might be useful for identifying things which are simultaneously pleasing to the intuition and wrong, they aren't "not even wrong". They do in fact contain meaningful (but either incorrect or non-rigorous) content

...although (sorry, I know this was supposed to be a TL:DR) I guess they might become "not even wrong" if they're foisted into models that don't have room for them. The free will debate made much more sense back when it was more "Can a person really control their destiny, or is that foolish hubris and the gods decide all?" and I think at some deeper emotional level many people who try to debate undissolved free will are really asking "Am I free or are the physics gods bossing me around?"...but then the ground shifted and the debate was inappropriately transferred forced into an arena where the gods were replaced by a lawful universe and now it's all complete nonsense unless we do that "redefine savory as umami" trick and say that free will is implemented by laws, that's right, the "God" was you all along! (I call it a "semantic trick" because it solves problems via a redefinition of word meaning, but respectfully: it's a semantic trick with a long, illustrious history spanning from the Upanishads to Lesswrong's own Yudkowski and it really works for unconfusing people. When a redefined meaning seems to carve reality better than the original, it's more correct in some respects.)

The original question about free will was a valid question, the original constructs of spirits and so-on not empty, even if attempts to engage then analytically often generate nonsense until you understood what exactly that fuzzy human intuition was attempting to communicate.

Reply
[-]Richard_Kennaway80

Suppose we had a word that described an optical illusion, but we didn't know it was an optical illusion. Later, we discover that it is an optical illusion, and the reality isn't like we thought. Don't you think our original word describing our naive perceptions continues to mean something?

For example, "sunrise".

Reply
[-]9eB160

The word savory long predates our modern understanding of umami. So when people use it, they could be using either the older, more general definition, or the newer, more specific definition. That is, before we knew that umami was a basic taste which represented glutamate receptors, other things that are related to umami would be considered part of the category of savory, like the experience of eating meat, for example. It seems a little bit silly that people chose to coopt an existing word to refer to a more specific phenomenon, but it's probably not the first time it has happened (although examples elude me at the moment).

Reply
[-]Richard_Kennaway90

(although examples elude me at the moment)

Heat. Energy. Atom. Planet. Star. Element. Sky. Life.

Reply
[-]Jiro50

although examples elude me at the moment

Theory.

(Although I was primed on this by having recently made a comment mentioning creationists.)

Reply
[-]Sniffnoy40

Hm; I was under the impression that, like, the old extension and the new extension were essentially the same -- pointing to the same cluster of things -- and the discovery of the glutamate receptor just clarified what it was exactly that that cluster had in common. I hadn't considered the possibility that the word was actually used more generally in the past.

Reply
[-]Jiro40

After he's finished congratulating you, explain that you have decided that it's almost certain, based on your priors, that we live in a simulation, and the being who runs this simulation is God. You call Him Fred. Most likely He's the super(universe) equivalent of a grad student.

I've always thought of a god as something that among other traits, is reasonably high in the hierarchy. An outside-universe grad student would not count, for the same reason that Dracula doesn't.

Furthermore, when most people say that God created the universe, by "the universe" they mean "everything except God himself". A simulation would not count as a "universe" even though we inherently have no way to perceive things outside it.

Reply
[-]CCC40

So, let me see if I understand this right - you're contending that human behaviour is constrained to behaviour that best benefits the genes? That is to say, that human goals are really sneakily disguised genetic goals?

Then what about people who take a vow of celibacy, who deliberately choose not to have descendants, and therefore subvert any possible genetic goal? How is that possible?

Reply
[-]PhilGoetz20

Well, for the particular case of vows of celibacy, let me put it this way:

  • Will you, personally, take a vow of celibacy?
  • How would you explain the reasoning of people who do take vows of celibacy?
  • Are there any errors or mistaken beliefs in their reasoning?

More generally, humans have values and heuristics that, taken together in the ancestral environment, usually increase genetic fitness. Sometimes they don't, especially when technology and a modern economy provides ability and incentives for people to add superstimuli to the environment. For example, the guy who plays so many video games that he never meets women, or the guy who drinks one Big Gulp(TM) a day until he weighs 300 pounds. Or the guy whose concern for his personal safety, combined with a belief in Heaven and Hell, convinces him to protect his long-term interests with a vow of celibacy.

Reply
[-]tavaton00

I don't want to be happy because that increases the likelihood that I'll have lots of happy, well-fed grandchildren. I want lots of happy, well-fed grandchildren because I believe that would increase the likelihood that I'm happy. My desire for grandchildren may change, especially if I think they would be unhappy. My desire for happiness will not change. This is not an error or a case of messed up priorities or me making decisions based on faulty assumptions, it's just how my values are.

Reply
[-]PhilGoetz00

These are values you consciously have. But the reason for all your values, if traced back far enough, lies in genetic fitness. Your not being conscious of that doesn't change it.

Reply
[-]tavaton40

The cause of my having these values doesn't really make a difference to what my values are. The values I'm conscious of is all that matters to me. I have no moral or emotional commitment whatsoever to reproductive efficiency. That is not one of my values.

Reply
[-]Richard_Kennaway00

But the reason for all your values, if traced back far enough, lies in genetic fitness. Your not being conscious of that doesn't change it.

But the reason for all your values, if traced back far enough, lies in atoms. Your not being conscious of that doesn't change it.

Reply
[-]PhilGoetz00

Atoms don't influence the evolution of a species or its values in any particular direction. The system we're looking at is bounded at "the bottom" by selection.

Reply
[-]EHeller00

Evolution is a super slow optimization process, and the environment changes. I'm equally "genetically fit" (i.e. I was born to survivors) to any thing alive today, but have a very different goal set.

Reply
[-]Vaniver00

I don't want to be happy because that increases the likelihood that I'll have lots of happy, well-fed grandchildren. I want lots of happy, well-fed grandchildren because I believe that would increase the likelihood that I'm happy.

It is not clear to me that you're using "happy" in the way that most people use "happy." Specifically, the things that people enjoy and the things that people pursue are often different, and whether or not people are pleased in near mode or far mode are different things. It could very well be that you pursue children but they decrease your enjoyment of life, and yet you pursue them despite knowing that.

Reply
[-]tavaton00

I use the formula engagement + meaning + enjoyment to calculate happiness. Children may well be a net negative to enjoyment at times. Whether or not I decide to have them depends on whether I believe that's going to be outweighed by engagement and meaning most of the time.

Reply
[-]Meni_Rosenfeld00

If your own celibacy somehow helps your relatives (who have a partial copy of your genes) reproduce, then the needs of your genes have been served. In general, genes have ways to pursue their agenda other than have their host reproduce. Sometimes genes even kill their host in an attempt to help copies of themselves in other hosts.

Reply
[-]skeptical_lurker60

Your own celibacy would have to massively benefit others - for instance, if you were going to have 2 children, your celibacy would have to allow your sibling to have 4 extra children.

Reply
[-]Meni_Rosenfeld00

Of course. I'm not recommending to any genes to have their host go celibate. I just disagree with the deduction "if you're ceilbate you can't have children, so there's no way your genes could benefit from it, QED".

Reply
[-]skeptical_lurker00

Well, it works in ants that share 75% of their genetic code with their siblings, but in humans... while its not impossible that celibacy could increase genetic fitness, its highly unlikely.

Reply
[-]Nornagest00

On average, that's true. But I've started wondering if maybe the standard gene-centered view doesn't take high variance in reproductive success into account.

Lately I've been reading some stuff by Napoleon Chagnon, who did a lot of (massively controversial) ethnography of the Yąnomamö people in the Sixties and Seventies. One of the things he tracked was the parentage of children in his area of study. It turns out there are enormous differences there, especially along the male line -- particularly prominent men might have as many as twenty or forty children, balanced out of course by correspondingly lower fertility among others. (The Yąnomamö are polygynous, and sometimes polyandrous, but for obvious reasons a wife with many husbands cannot produce as many children as a husband with many wives.)

If that sort of thing's typical of the EEA (a hypothesis, again, that's massively controversial), then it's not impossible that there could have evolved behavioral adaptations diminishing potential fertility, if they kicked in under conditions of expected low fertility -- the downside loss is low enough that aid to your siblings or cousins could make up for it. That'd be consistent with cultural traditions like younger sons going into the clergy.

Reply
[-]skeptical_lurker00

That'd be consistent with cultural traditions like younger sons going into the clergy.

Do younger sons have extremely low evo fitness?

I think I'd have to see some really strong evidence of celibacy increasing genetic fitness, otherwise I'm just going to assume this is a case of memes overpowering genes. There could of course, for example, be a gene for religiousness that generally increases evo fitness, but occasionally causes someone to take a vow of celibacy. This would on average increase evo fitness.

I don't know how exactly genes translate into behaviour. But I would guess that its possible for genes to code for religiousness, but coding for or against a religious vow of celibacy is impossible, as this is far too specific a behaviour.

Reply
[-]Nornagest00

Do younger sons have extremely low evo fitness?

That's extremely culture-specific. I was talking about cultures like medieval European nobility, where you'd typically want an heir and a spare, both of whom you'd go to some effort to marry off, but where subsequent sons just got in the way and often ended up in the clergy. Other cultures would have their own analogous situations.

At the gene level, what I'm talking about wouldn't look like something that universally codes for a +kin, -self fertility tradeoff regardless of environment; for more or less the reasons you gave in the grandparent that's quite unlikely to be selected for. But if you throw environmental interactions into the mix you can do more interesting things. One pathway for example might start by modulating hormone levels in response to mate availability, for example to encourage higher-risk mating strategies in rich environments. That gives the biochemistry something it can work with, and opens up more complicated strategies, some of which might include kin-centric strategies gated by specific hormone levels.

Even that isn't going to look like a gene coding for a vow of celibacy. But it might translate to a vow of celibacy given a certain cultural context and personal history.

Reply
[-][anonymous]40

She'd begun with the postulate that intelligence was unexplainable, perhaps because that was what she needed God for.

Don't be mean: she began with the postulate that intelligence is inexplicable because firstly, she cannot explain it, and secondly, most purported competent professionals in psychology and philosophy of mind cannot explain it, either.

Neither school would propose that human terminal values contain anything we, as modern humans, value.

In which case you are simply defining the issue wrongly. Evolution was not a very careful designer: we are not Evolution!Friendly. Our cognitive algorithms do not compute anything that resembles "inclusive genetic fitness" except by coincidence.

Evolution once "trusted" that any creature that accidentally optimized for something other than inclusive genetic fitness would die out (ie: selection pressures would operate more quickly than any mere creature could optimize the environment). Well too bad: we think, act, and change far faster than the timescales sufficient for evolutionary pressures to work.

But of course, if you really believe this sort of thing, I can always just rewrite you to go along with some different set of values of my own devising. After all, it's not like you had any real goals or values of your own, right? So what have you lost as you let me fiddle around in your mind, that wasn't an empty category in the first place?

Nice job proving your own suicide completely rational /s -- but I'm sure you got some really scary existential willies out of that, which seems to be what some people like about their personal form of "rationality".

Reply
[-]PhilGoetz10

Our cognitive algorithms do not compute anything that resembles "inclusive genetic fitness" except by coincidence.

This is very wrong. But to address the point you're trying to make: This is like saying that the computer which controls a robot arm doesn't compute anything that resembles arm motions; it just computes numbers. Those numbers, sent into the arm actuators, produce motions. Your cognitive algorithms, executed in a human body, maximize its inclusive genetic fitness. "You" don't need to be aware that that's what it's doing, nor do the algorithms need to represent the concept of genetic fitness.

After all, it's not like you had any real goals or values of your own, right? So what have you lost as you let me fiddle around in your mind, that wasn't an empty category in the first place?

I have feelings, which I enjoy. "I" have values which are not aligned with the values of my genes. But they aren't terminal values in the sense that may be required by Friendly AI theory.

One of the many problems with FAI theory, as Eliezer has written about it, is that you want to "improve" human cognition. This involves looking at things humans do and deciding which things are values, and which are errors, so you can eliminate the errors but keep the values. But I don't know any way to do this other than tracing all the values back to the terminal values, keeping those, and throwing out the instrumental values. But it turns out that everything we, the conscious riders on our physical bodies, value, is instrumental.

Reply
[-][anonymous]30

Your cognitive algorithms, executed in a human body, maximize its inclusive genetic fitness.

No, they don't. They simply, straightforwardly don't. Eating sugar instead of fish, or using birth control, browsing the web (when I could be having sex to increase my count of offspring), and remaining loyal to my fiancee (when I could plainly get better stock than her should I sincerely attempt to maximize the count and fitness of my offspring) are all behaviors generated by my cognitive algorithms that straightforwardly seek goals other than inclusive genetic fitness (in this case: two sensual pleasures, one intellectual one, and a combination of pair-bonded attachment and keeping up the moral trust in our relationship).

I have feelings, which I enjoy. "I" have values which are not aligned with the values of my genes.

There is no need for scare-quotes: just because your individuality does not correspond to an immortal, supernatural soul doesn't mean it corresponds to nothing at all.

But they aren't terminal values in the sense required by Friendly AI theory.

In which case it is the theory that requires correction, not the lower-level (lower-level in the Hierarchical Bayesian sense, closer to the evidence) belief that we have values.

One of the many problems with FAI theory, as Eliezer has written about it, is that you want to "improve" human cognition.

Actually, the chief problem with FAI theory as written by Eliezer is that there simply isn't much of it!

This involves looking at things humans do and deciding which things are values, and which are errors, so you can eliminate the errors but keep the values. But I don't know any way to do this other than tracing all the values back to the terminal values, keeping those, and throwing out the instrumental values.

But it turns out that everything we, the conscious riders on our physical bodies, value, is instrumental.

Our values are only instrumental from the point of view of evolution. That's not an objective point of view: in order to decide that our values are heuristics for maximizing inclusive genetic fitness, you first have to assume very naive definitions of "terminal" (something like: goal of the "least-caused" optimization process) and "instrumental" (something like: goal of a "more-caused" optimization process).

The issue is, of course, that humans don't particularly care (or, much of the time, know) what caused us, and also that locating evolution as the least-caused optimizer is incorrect: entropy is the least-caused optimizer (in fact, it's the only elemental force of optimization in the universe: it drives the arrow of time).

Even this already gives us a way to straightforwardly measure which goals and values are terminal, and which instrumental: the precise causal processes underlying an instrumental goal are a matter of great evaluative import to our human cognitive algorithms, whereas the precise causal processes underlying a terminal goal are a matter of no import at all. When you stop caring how your goal/value/feeling got there, and only care about fulfilling it, you've found a terminal goal/value/feeling.

To take a common example: love! It's true, terminal love when you simply don't give half a damn how it's implemented!

Now, to get further on this I'll need a decent theory of how conceptual and causal/generative abstraction take place in humans, how we get from "chair" to "big cloud of buzzing probability clouds of quarks" and back again, but that kind of progress on human cognitive algorithms and evaluative judgements will give us a solid way to talk about terminal versus instrumental: when the details at the lower-level of reality can be thrown out without altering the evaluative judgement, you've found something that is terminally relevant, and from which value/relevance/usefulness/utility flows backwards into other things during the probabilistic backwards-chaining process the human mind seems to use for planning.

Once again: just because your human concepts do not correspond to objects at the most ontologically basic and causally early levels of reality does not mean they fail to correspond to anything. IMNSHO, if you can find it in yourself to assent to chair realism, you should equally assent to value realism: both those concepts correspond to real things, even if "values" needs reforming (a change in the induced conceptual definition needed to correspond to the appropriate data) from its earlier "innate feature of the world" formulation in order to do it.

Reply
[-]PhilGoetz20

Your cognitive algorithms, executed in a human body, maximize its inclusive genetic fitness.

No, they don't. They simply, straightforwardly don't.

They maximize fitness the way any optimization method maximizes a complex function: unreliably, slowly, not always moving in the right direction. All that is required to say that something "maximizes" a function is that it generally increases its value. Perhaps "optimizes" would be a better word.

In some cases today, these heuristics no longer optimize fitness at all. As we all know. This is not a point worth dwelling on.

There is no need for scare-quotes: just because your individuality does not correspond to an immortal, supernatural soul doesn't mean it corresponds to nothing at all.

The quotes are there because resolving what "I" refers to is non-trivial, and the discussion here depends on it.

In which case it is the theory that requires correction, not the lower-level (lower-level in the Hierarchical Bayesian sense, closer to the evidence) belief that we have values.

I never said we don't have values. I said human values aren't terminal values. You need to make sure you understand that distinction before criticizing that part of my post.

Actually, the chief problem with FAI theory as written by Eliezer is that there simply isn't much of it!

Agreed.

Our values are only instrumental from the point of view of evolution. That's not an objective point of view:

Yes, it is. The terminal values are what the system is optimizing. What the system optimizes doesn't depend on your perspective; it depends on what provides feedback and error-correction. Reproductive success is the feedback mechanism; increasing it is what the system develops to do. Everything above is variable, inconstant, inconsistent; everything below is not being optimized for.

also that locating evolution as the least-caused optimizer is incorrect: entropy is the least-caused optimizer

See above. The feedback to the human system occurs at the level of reproductive fitness. What you just said implies that humans actually maximize entropy. Think about that for a few moments. I mean, technically, we do; everything does. But any intelligent analysis would notice that humans reduce entropy locally.

When you stop caring how your goal/value/feeling got there, and only care about fulfilling it, you've found a terminal goal/value/feeling.

Try enumerating examples of terminal values. You'll find they are contradictory, they change within individuals and within societies rapidly, they are not constants of human history, and they are very often things that one would think we would rather eliminate from society than build a big AI to guarantee we will have them with us forever. Perhaps more importantly, the "biases" that LessWrong was founded to eliminate are indistinguishable from those kinds of values. See Human errors, human values.

when the details at the lower-level of reality can be thrown out without altering the evaluative judgement, you've found something that is terminally relevant, and from which value/relevance/usefulness/utility flows backwards into other things during the probabilistic backwards-chaining process the human mind seems to use for planning.

First, not many values above the level of the genetic remain constant across time and across the Earth.

Second, that wouldn't help with resolving conflicts between higher "instrumental" values. If you removed the instrumental values, leaving the low-level judgements, and used greater computational power to optimize them more accurately, the human would produce different outputs. Would the human then have been "debugged" because it produced outputs more in accordance with the "terminal" level? Why should low-level judgements like galvanic skin response have precedence over cognitive judgements? The things that you would list as "terminal values" would tend to be things we have in common with all mammals. "Human values" should include some values not also found in dogs and pigs. But evolution very often works by elaboration, and it would not be surprising if most or all of the "human" part of our values were in things layered on top of these "terminal" values.

Third, there is no way to distinguish values from mistakes/biases.

Fourth, there is probably no way to "extrapolate" values away from the organism. Your list of "terminal human values" would be full of statements like "Humans value sweet and salty tastes" and "Males value having their penises stroked." This is not, I think, what is most-important for us to pass on to the Universe a billion years from now. They will not apply to non-human bodies. Any attempt by an AI to enforce these values would seem to require keeping the standard human body for the rest of the life of the Universe.

just because your human concepts do not correspond to objects at the most ontologically basic and causally early levels of reality does not mean they fail to correspond to anything

I don't think that relates to anything I wrote.

Reply
[-][anonymous]40

First of all, let me say that I've been busy today and thus apologize for the sporadic character of my replies. Now, to begin with the most shocking and blunt statements...

Fourth, there is probably no way to "extrapolate" values away from the organism. Your list of "terminal human values" would be full of statements like "Humans value sweet and salty tastes" and "Males value having their penises stroked."

What's the problem? Were you expecting something other than humanity to come through in your model of humanity? Your phrasing signals that you are looking down on both sex and the enjoyment of food, and that you view them as aesthetically and/or morally inferior to... what? To "nonhuman bodies"? To intellectual pursuits?

Do you think intellectual pursuits will not also have their place in a well-learned model of human preferences? Are you trying to signal some attachment to the Spiral instinct/will-to-power/tsuyoku naritai principle? But even if you terminally value the expansion of your own causal or optimization power, there are other things you terminally value as well; it is unwise to throw away the rest of your humanity for power. You'll be missing out.

To repeat one of my overly-repeated catch phrases: cynicism and detachment are not innately virtuous or wise. If what real, live human beings actually want, in the limit of increasing information and reflection, is to spend existence indulging tastes you happen to find gauche or déclassé, from where are you deriving some kind of divine-command-style moral authority to tell everyone, including yourself, to want things other than what we actually want?

What rational grounds can you have to say that a universe of pleasures - high and low - and ongoing personal development, and ongoing social development, and creativity, and emotionally significant choices to make, and genuine, engaging challenges to meet, and other people to do it all with (yes I am just listing Fun Theory Sequence entries because I can't be bothered to be original at midnight)... is just not good enough for you if it requires learning a different way to conceptualize it all that turns out to correspond to your original psychological structure more than it corresponds to a realm of Platonic Forms, since there turned out not to be Platonic Forms?

Why do you feel guilty for not getting the approval of deities who don't exist?

Any attempt by an AI to enforce these values would seem to require keeping the standard human body for the rest of the life of the Universe.

Or, and this is the neat bit, to create new kinds of nonhuman bodies, or nonbodily existence, that are more suited to what we value than our evolved human ones.

This is not, I think, what is most-important for us to pass on to the Universe a billion years from now.

Simply put: why not?

Try enumerating examples of terminal values. You'll find they are contradictory, they change within individuals and within societies rapidly, they are not constants of human history, and they are very often things that one would think we would rather eliminate from society than build a big AI to guarantee we will have them with us forever.

Again: this is why we are trying to reduce the problem to cognitive algorithms, about which facts clearly exist, rather than leaving it at the level of "a theory is a collection of sentences written in first-order logic augmented with some primitive predicates". The former is a scientific reality we can model and compute with, while the latter is a cancerous bunch of Platonist nonsense slowly killing the entire field of philosophy by metastasizing into whole fields and replacing actual reductionist rigor with the illusion of mathematical formalism.

(The above is, of course, a personal opinion, which you can tell because of the extreme vehemence. But holy shit do I hate Platonism and all its attendant fake rigor.)

Anyway, the rest I'll have to answer in the morning, after a night's sleep.

Reply
[-]Vaniver10

I am having difficulty seeing what you don't understand about PhilGoetz's point. You read like you're reacting to overstatements on his part, but it looks to me like you're reaching much further from reality than he is, or uncharitably interpreting his statements.

We can abstract from our values to principles, and so on, but what makes the difference between an instrumental value and a terminal value is that a terminal value is one that exists for its own sake. Inclusive genetic fitness does match that definition, because natural selection is a thing that slowly replaces things with lower inclusive genetic fitness with things with higher inclusive genetic fitness. This is what biologists mean by 'maximize,' and it's different from what numerical optimization / math people mean by 'maximize.'

Is it true that you are doing the most you can to maximize your inclusive genetic fitness (IGF)? No, you're clearly suboptimal. But it is clearly true that your ancestors reproduced, and thus your genes are a product of the evolutionary project to gradually replace lower IGF with higher IGF, and in that sense you are doing more on average to increase your IGF than the counterfactual yous that do not exist because their ancestors failed to reproduce. That seems to be what PhilGoetz is arguing for on the object level (and he should correct me if that's not the case.)

So now we take a step back to talk about values. When we look at possible values, we see lots of things that want to exist for their own sake (view how people talk about truth, justice, equality, and so on), but humans only seem to desire them because of their effects (view how people act about truth, justice, equality, and so on). It looks like people choose baskets of values and make tradeoffs between them- but in order to make tradeoffs between two instrumental values, there must be some terminal value that can look at options and say "option A is better than option B."

It looks like the historical way this happens is that people have values, and some people reproduce more / spread their memes more, and this shifts the gene and meme (here, read as "value") distributions. Broadly, it seems like genes and memes that are good for IGF are somewhat more popular than genes and memes that are bad for IGF, probably for the obvious reason.

That is, it looks like the universe judges value conflicts by existence. If there are more Particularists than Universalists, that seems to be because Particularists are out-existing the Universalists. To the extent that humans have pliable value systems, they seem to look around, decide what values will help them exist best, and then follow those values. (They also pick up their definition of "exist" from the environment around them, leading to quite a bit of freedom in how humans value things, though there seem to be limits on pliability.)

Moving forward, we seem to have some control over how the economic and military frontiers will change, and thus some control over what values will promote more or less existence. We probably want to exert that control in order to ensure the 'right' morality is favored.

But... if the practical determines the moral, and we want to decide what is practical using the moral, we now have a circular situation that it's difficult to escape.

Our deeply held values are not "deeply held" in the sense that we can go meta and justify them to someone who doesn't have them, but does share our meta-level value generating process. If we put a hypothetical twin you into a Comanche tribe to be raised, and then once he reached your current age you and he tried to come up with the list of human values and optimal arrangement of power, there would probably be significant disagreement. So PhilGoetz is pessimistic about a plan that looks at humans and comes up with the right values moving forward, because the system that determines those values is not a system we trust.

Reply
[-][anonymous]20

We can abstract from our values to principles, and so on, but what makes the difference between an instrumental value and a terminal value is that a terminal value is one that exists for its own sake.

"Sakes" are mental concepts. Reality does not contain extra-mental sakes to exist for.

Inclusive genetic fitness does match that definition, because natural selection is a thing that slowly replaces things with lower inclusive genetic fitness with things with higher inclusive genetic fitness.

Again: since evolution does not have a mind, I don't see how you could label inclusive genetic fitness as "terminal". It is the criterion for which evolution optimizes, but that's not nearly the same thing as a "terminal value" in any ethical or FAI sense.

(And, as I mentioned, it is very definitely not terminal, in the sense that it is a sub-optimizer for the Second Law of Thermodynamics.)

Is it true that you are doing the most you can to maximize your inclusive genetic fitness (IGF)? No, you're clearly suboptimal. But it is clearly true that your ancestors reproduced, and thus your genes are a product of the evolutionary project to gradually replace lower IGF with higher IGF, and in that sense you are doing more on average to increase your IGF than the counterfactual yous that do not exist because their ancestors failed to reproduce. That seems to be what PhilGoetz is arguing for on the object level (and he should correct me if that's not the case.)

While your statement about my being more genetically "fit" than, say, the other sperm and egg cells that I killed off in the womb is entirely correct, that has basically nothing to do with the concept of "terminal values", which are strictly a property of minds (and which evolution simply does not have).

So now we take a step back to talk about values. When we look at possible values, we see lots of things that want to exist for their own sake (view how people talk about truth, justice, equality, and so on), but humans only seem to desire them because of their effects (view how people act about truth, justice, equality, and so on). It looks like people choose baskets of values and make tradeoffs between them- but in order to make tradeoffs between two instrumental values, there must be some terminal value that can look at options and say "option A is better than option B."

Or a person must simply trade off their terminal values against each-other, with some weighting deciding the final total utility.

It seems to me like we need a word to use in Evaluative Cognitive Algorithms Theory other than "values", since people like you and PhilGoetz are confusing "values" in the Evaluative Cognitive Algorithms Theory sense of the word with "values" in the sense of what a non-naturalist ethicist or a politician talks about.

Moving forward, we seem to have some control over how the economic and military frontiers will change, and thus some control over what values will promote more or less existence. We probably want to exert that control in order to ensure the 'right' morality is favored.

If you are thinking in terms of promoting the "right" morality in an evolutionary sense, that the "right" morality is a program of which you "must" make copies, you are not using the term "morality" in the sense that Evaluative Cognitive Algorithms Theory people use it either.

(And certainly not in any sense that would invoke moral realism, but you don't seem to have been claiming moral realism in the first place. On a side note, I think that trying to investigate what theories allow you to be "realist about X" is a useful tool for understanding what you mean by X, and morality is no exception.)

But... if the practical determines the moral, and we want to decide what is practical using the moral, we now have a circular situation that it's difficult to escape.

No we don't. One optimizer can be stronger than another. For instance, at this point, humanity is stronger than evolution: we are rapidly destroying life on this planet, including ourselves, faster than anything can evolve to survive having our destructive attentions turned its way. Now personally I think that's bloody-stupid, but it certainly shows that we are the ones setting the existence pressures now, we are the ones deciding precisely where the possible-but-counterfactual gives way to the actual.

And unfortunately, we need modal logic here. The practical does not determine what the set moral modality we already possess will output. That modality is already a fixed computational structure (unless you're far more cultural-determinist than I consider reasonable).

Our deeply held values are not "deeply held" in the sense that we can go meta and justify them to someone who doesn't have them, but does share our meta-level value generating process.

I am confused by what you think a "meta-level value-generating process" is, or even could be, at least in the realms of ethics or psychology. Do you mean evolution when you say "meta-level value generating process"?

And additionally, why on Earth should we have to justify our you!"values" to someone who doesn't have them? Seeking moral justification is itself an aspect of human psychology, so the average non-human mind would never expect any such thing.

If we put a hypothetical twin you into a Comanche tribe to be raised, and then once he reached your current age you and he tried to come up with the list of human values and optimal arrangement of power, there would probably be significant disagreement.

There would be a significant difference in preference of lifestyles. Once we explained each-other to each-other, however, what we call "values" would be very, very close, and ways to arrange to share the world would be invented quite quickly.

(Of course, this may simply reflect that I put more belief-weight on bioterminism, whereas you place it on cultural determinism.)

Reply
[-]Vaniver00

Again: since evolution does not have a mind, I don't see how you could label inclusive genetic fitness as "terminal".

Perhaps it would be clearer to discuss "exogenous" and "endogenous" values, as the relevant distinction between terminal and instrumental values are that terminal values are internally uncaused, and instrumental values are those pursued because they will directly or indirectly lead to an improvement in those terminal values, and this maps somewhat clearly onto exogenous and endogenous.

That is, of course this is a two-place word. IGF is exogenous to humans, but endogenous to evolution (and, as you put it, entropy is exogenous to evolution).

So my statement is that we have a collection of values and preferences that are moderately well-suited to our environment, because there is a process by which environments shape their inhabitants. As we grow more powerful, we shape our environment to be better suited to our values and preferences, because that is how humans embody preferences.

But we have two problems. First, our environment is still shaping our values and preferences, and thus the sort of world that we most want to live in might not be a world that would be mostly populated by us. Second, if we have any conflicts about preferences, typically we would go up a level to resolve those conflicts--but it is obvious that the level "above" us doesn't have any desirable moral insights. So we can't ground our conflict-resolution process in something moral instead of practical.

Of course, this may simply reflect that I put more belief-weight on bioterminism, whereas you place it on cultural determinism.

It seems to me that near-mode values are strongly biodetermined, but far-mode values are almost entirely culturally determined. Since most moral philosophy takes place in far mode, cultural determination is far more relevant. You and your Comanche twin might be equally anxious, say, but are probably anxious about very different things and have different coping strategies and so on.

ways to arrange to share the world would be invented quite quickly.

I picked Comanche specifically because they were legendary raiders with a predatory morality.

Reply
[-][anonymous]00

First, our environment is still shaping our values and preferences, and thus the sort of world that we most want to live in might not be a world that would be mostly populated by us.

I simply have to ask: so what? I place no particular terminal value on evolution itself. I see nothing wrong, neither aesthetically nor morally, with simply overriding evolution through human deeds, the better to create the kind of world that, indeed, we living humans most want to live in. Who cares how probable it was, a priori, that evolution should spawn our sort of people in our preferred sort of environment?

Well, I suppose you do, for some reason, but I'm really confused as to why.

Second, if we have any conflicts about preferences, typically we would go up a level to resolve those conflicts

Actually, I disagree: we usually just negotiate from a combination of heuristics for morally appropriate power relations (picture something Rawlsian, and there are complex but, IMHO, well-investigated sociological arguments for why a Rawlsian approach to power relations is a rational idea for the people involved) and morally inappropriate power relations (ie: compulsion and brute force).

I suppose you could call the former component "going up a level", but ultimately I think it grounds itself in the Rawls-esque dynamics of creating, out of social creatures who only share a little personality and experience in common among everyone, a common society that improves life for all its members and maximizes the expected yield of individual efforts, particularly in view of the fact that many causally relevant attributes of individuals are high-entropy random variables and so we need to optimize the expected values, blah blah blah. Ultimately, human individuals do not enter into society because some kind of ontologically, metaphysically special Fundamental Particle of Morals collides with them and causes them to do so, but simply because people need other people to help each-other out and to feel at all ok about being people -- solidarity is a basic sociological force.

So we can't ground our conflict-resolution process in something moral instead of practical.

As you can see above, I think the conflict-resolution process is the most practical part of the morals of human life.

It seems to me that near-mode values are strongly biodetermined, but far-mode values are almost entirely culturally determined. Since most moral philosophy takes place in far mode, cultural determination is far more relevant.

Frankly, I think this is just an error on the part of most so-called moral philosophy, that it is conducted largely in a cognitive mode governed by secondary ideas-about-ideas, beliefs-in-beliefs, and impressions-about-impressions, a realm almost entirely without experiential data.

While I don't think "Near Mode/Far Mode" is entirely a map that matches the psychological territory, insofar as we're going to use it, I would consider Near Mode far more morally significant, precisely because it is informed directly by the actual experiences of the actual individuals involved. The social signals that convey "ideas" as we usually conceive of them in "Far Mode" actually have a tiny fraction of the bandwidth of raw sensory experience and conscious ideation, and as such should be weighted far more lightly by those of us looking to make our moral and aesthetic evaluations on data the same way we make factual evaluations on data.

The first rule of bounded rationality is that data and compute-power are scarce resources, and you should broadly assume that inferences based on more of each are very probably better than inferences in the same domain performed with less of each -- and one of these days I'll have the expertise to formalize that!

Reply
[-]Vaniver00

I simply have to ask: so what? I place no particular terminal value on evolution itself. I see nothing wrong, neither aesthetically nor morally, with simply overriding evolution through human deeds, the better to create the kind of world that, indeed, we living humans most want to live in.

I don't think I was clear enough. I'm not stating that it is value-wrong to alter the environment; indeed, that's what values push people to do. I'm saying that while the direct effect is positive, the indirect effects can be negative. For example, we might want casual sex to be socially accepted because casual sex is fun, and then discover that this means unpleasant viruses infect a larger proportion of the population, and if they're suitable lethal the survivors will, by selection if not experience, be those who are less accepting of casual sex. Or we might want to avoid a crash now and so transfer wealth from good predictors to poor predictors, and then discover that this has weakened the incentive to predict well, leading to worse predictions overall and more crashes. Both of those are mostly cultural examples, and I suspect the genetic examples will suggest themselves.

That is, one of the ways that values drift is the environmental change brought on by the previous period's exertion of their morals may lead to the destruction of those morals in the next period. If you care about value preservation, this is one of the forces changing values that needs to be counteracted or controlled.

Reply
[-]Wes_W20

The traits of evolved organisms are usually a reasonable approximation of fitness-maximizing, because they're the output of billions of years of a fitness-maximizing process; I'm with you on not being willing to call it "coincidence". But this seems silly:

Your cognitive algorithms, executed in a human body, maximize its inclusive genetic fitness.

My cognitive algorithms are the results of a process which maximizes inclusive genetic fitness. This does not mean these algorithms themselves maximize that.

On the margin, a large fraction of the outputs of this process definitely don't maximize fitness. Otherwise, there would be nothing for selection to eventually weed out!

Reply
[-]PhilGoetz20

My cognitive algorithms are the results of a process which maximizes inclusive genetic fitness. This does not mean these algorithms themselves maximize that.

I'm using "maximize" loosely. I agree with your observations.

Reply
[-]Viliam_Bur30

Seems to me we have a few logical jumps here.

First, yes, alleles play zero-sum games against each other. So what? At this moment, some alleles are winning and some alleles are losing; some of them have higher percentage in the gene pool, some have lower. So, why not take the snapshot of what the gene pool is now, and define the human values accordingly. (And iterate. If the current values prefer to have some alleles removed, then remove them, and define the human values according to the remaining ones.)

Second, yes, human behavior and preferences change depending on the environment. For example, when humans are hungry, they may prefer to eat even when it means killing another person; but when they are not hungry, they may prefer to live in peace with other humans. So which one is the true preference? Well, humans also have preferences about environment, e.g. they prefer not being hungry to being hungry. So we can try to iterate these values and environments together, and find out that the attractor is the situation where humans are not hungry and live in peace with other humans. (Technically, there could be multiple attractors.)

The facts that alleles are playing zero-sum games, and that human preferences depend on the environment, do not per se make the task of finding human "extrapolated volition" impossible.

Reply
[-]Aardvark20

"Virtue" is a category;..."
I do not know what meaning you intended to convey here. The capital 'V' could be the result of beginning a sentence or it could be reification - the raising of virtue to the level of a moral absolute.

I have learned to beware terms with capital letters; Morality, Altruism, Truth, Spirit, Noble... each of these gets used by those who adopt a Neo-Platonist theory of mind.

[In Plato's parable of the Cave, mind is where the category of the universe intersects his (invented) category of Forms.]

It is simple enough to comprehend the 'Form of the Good' but it must remain an empty category because anything I try to put there fails to define 'good'; I may put the virtue of honesty in the highest position but it does not account for the virtues of kindness or valour. Honesty is modified by other virtues so it cannot be the Absolute.

I keep encountering the word Truth, capital 'T', used as if it were a moral absolute independant of mind and the universe.

When this mind-set are challenged to define 'Truth' the authors hold that, since I find nothing in that category, I have no such category, thereby reinforcing their notion that we do not comprehend it.

In their mind it comes down to "I get it by faith" and, by extension, "you don't get it because you have no faith."

Simply asking for evidence or definitions can serve to reinforce their cognitive bias.

It is important, to me at least, to think in terms of empty categories because I can then say that I DO believe in the Truth, without comiting myself to anything others may try to put in that category.

I similarly believe in 'nobility' because that is the means by which I frame the question "what is nobler in the mind...?"

My former brethren are the sort who use the phrase "what would Jesus do?" - but I came to realise that I was in fact usin that question in the same way as Hamlet; seeking my own response and calling that 'spiritual' when, in fact, it is scanning one's subconscious/memory. We search for a pattern which must both

  1. relate to the current problem and
  2. ensure one remains the hero in that story.

Thus, for example, when I confronted the main leader of my former church on gay rights issues, rather than trying to understand something new he reached for the explanation/the story that "anything I do not understand is spirit, therefore this is a spirit speaking to me. It is not nice, like the Holy Spirit, so it must be an evil spirit..."

In his view I still have a demon and he and his followers either do their best to cast it out, mock me or avoid me altogether. After all, it is human nature, from childhood, to avoid becoming the villain in one's own narrative. We tend to throw tantrums or sulk when we're caught out.

To conclude; having a naturalistic explanation of mind as a subset of the universe, does not mean we do not have categories for what could, without evidence, be beyond the universe. We even retain categories when we realise they’re evidently empty, like the category ‘Intelligent Design’. One can only hope for a future humanity in which that category is no longer an idol.

Reply
[-]CCC00

"anything I do not understand is spirit"

Well, that's silly of him. I do not understand quantum physics, this does not mean that quantum physics is spirit.

(This sounds like a fully general counterargument, which has its own problems...)

Reply
[-]TheAncientGeek20

Non-computational consciousness

See John Searle's Chinese room argument. Or don't. It defines consciousness as an impossible uncategory, arguing that anything that can be understood obviously can't be conscious.

  1. That doesnt oviously fulfill the two criteria of categoreal problermaticness you mentioned above -- necessary emptiness, and unobservability. It seems that you are implicitly adding a third criterion of Incomprehensibility. But, as a naturalist, you dont have a guarantee that everything is comprehnsible to your monkey brain, including your bain itself.

  2. The other two citeria are problematic as well. Observability? I've met people who disbelieve in matter because it..apart from its properties and behavioiur..is unobservable.

  3. The instruction to read Searle's Minds and Machines is puzzling. For one thing it is about semantics , not consciousness.And it is supposed to weigh against computational semantics, or maybe consciousness, if that is an adequate synonym for sematics.

  4. ..and you seem to be arguing for a reductionistic undestanding of consciousness, not a computational one, which is something much more specific.

  5. ..and reductionism is tricky. If you understand things by breaking them down into components, you are going to end up with ultimate components you don't understand. But you object to incomprehensible categories, or categories of the incomprehesible, or something.

  6. . ..and consciousness is tricky. If you go around telling people that cosnciousness is computation, dont be surprised if they ask you how to code SeeRed().

Reply
[-]Epictetus20

Humans are genes' way of reproducing themselves. Human behavior implements a utility function all of whose terminal values are statements about gene allele frequencies. As humans were not even aware of allele frequencies until recently, anything a human thinks it values cannot be a terminal value of a human utility function.

I don't find this obvious. Genes may have the goal of reproducing themselves and certain genes have adopted a strategy of using humans to do so. If we model human behavior by a utility function, it makes sense to assert that this would depend on the genes--if it didn't, it's unlikely humans would have evolved in the first place. So far, so good.

The question is whether the interactions of ~20,000 genes among themselves and with the environment doesn't add anything extra. Figure that genes are really just elementary particles arranged in a certain configuration, yet the gene's quest for reproduction does not have a clear origin from the elementary particles themselves. It's worth asking whether scaling up from genes to humans has its own emergent phenomena, where a human has values that don't obviously correspond to those of the genes.

Reply
[-]XerxesPraelor10

Try this experiment on a religious friend: Tell him you think you might believe in God. Then ask him to list the qualities that define God.

Before reading on, I thought "Creator of everything, understands everything, is in perfect harmony with morality, has revealed himself to the Jews and as Jesus, is triune."

People seldom start religions by saying they're God. They say they're God's messenger, or maybe God's son. But not God. Then God would be this guy you saw stub his toe, and he'd end up like that guy in "The Man Who Would Be King."

That's what's so special about Christianity - Jesus is God, not just his messenger or Son. The stubbed toe problem isn't original, it comes up in the Gospels, where people say "How can this be God? We know his parents and brothers!"

PPE: I see you added a footnote about this; still, even in the OT God lets himself be argued with - that's what the books of Job and Habakkuk are all about. Paul also makes lots of arguments and has a back and forth style in many books.

A belief in the God that is an empty category is wrong, but it's misrepresenting religion (Judeo-Christian ones in particular) to say that all or even most or even a substantial minority of its adherents have that sort of belief.

But if for some reason you want to know what "human terminal values" are, and collect them into a set of non-contradictory values, ethics gets untenable, because your terminal values benefit alleles, not humans, and play zero-sum games, not games with benefits to trade or compromise.

Evolution isn't perfect - the values we have aren't the best strategies possible for an allele to reproduce itself, they're only the best strategies that have appeared. This leaves room for a difference between the good of a person and the good of their genes. Thou art Godshatter is relevant here. Again, just because our human values serve the "values" of genes doesn't mean that they are subject to them and are somehow turned "instrumental" because evolution was the reason why they developed.

Reply
[-]gjm20

I thought "Creator of everything [...]"

Of the qualities you list, the last two -- the specifically Christian ones -- mean that if you really take this as defining "God" then you have to say that, e.g., Muslims do not believe in God (because they are very insistent that he is not triune) and nor do any monotheists who don't come from the "Abrahamic" tradition. Of course you are at liberty to define "God" that way, but it doesn't look to me like a great idea.

If we drop those (considering them as defining "Christianity" rather than "God", or something), and also (in accordance with Phil's stipulation, though I'm not sure it's a fair one) throw out "is in perfect harmony with morality" on the grounds that Phil doesn't like the term "perfect", we're left with "Creator of everything, understands everything". I think "Fred" satisfies the first condition but not the second.

So I think Phil's thought experiment is a bit broken. Maybe we can salvage it. Suppose now that our universe was indeed created by a graduate student in some other universe; his mental capacities far exceed ours and he does in fact understand everything that goes on in our universe. And let's suppose our moral values were somehow implanted in us by this graduate student in accordance with his own.

Does the grad student now fit your first three criteria? If so, would you call him God?

(My guess: he doesn't, because if there are universes other than ours then creating just our universe doesn't for you count as the right sort of thing to be God. That seems reasonable, though I remark that e.g. the creation story in Genesis 1 seems to show God already having some raw material to work with before he gets started creating; you probably don't want to say that whoever wrote Genesis 1 didn't believe in God.)

The stubbed toe problem isn't original, it comes up in the Gospels

What the onlookers are complaining at there isn't that Jesus is claiming to be God. It's that he's claiming to have come down from heaven. It's far from clear that Jesus actually claimed to be God -- there are a couple of places in the gospels where he says things that can be taken that way, but they're never fully explicit and there's no guarantee that the authors of the gospels exactly reproduced what he said.

Paul also makes lots of arguments

He does. But so far as I can recall, he always seemed pretty certain he was correct. There's that one time where he takes care to qualify two things he says as (1) "not I, but the Lord" and (2) "I, not the Lord", so clearly he admits his own fallibility in principle, but is there any instance where he admits having taught something wrong or says in so many words that something he's teaching might be wrong?

I do think what Phil says about "God as the ultimate uncategory" is ... less than perfectly fair. (I also think Phil is being less than perfectly clear about the distinction between "category C is defined negatively" and "category C is defined negatively to make it immune from criticism".) But I do also think he has a point: religious doctrines tend to be unfalsifiable, when evidence does come along they get reinterpreted more often than jettisoned, etc.

(In fact, the best example known to me of the sort of process I think Phil is describing comes from the early days of Christianity, when the idea of the Trinity was being worked out. What they ended up with was a situation where basically anything at all definite and comprehensible you might say about the Trinity has at one time or another been officially declared heretical. So you have a doctrine you're required to believe but forbidden to make sense of.)

Reply
[-][anonymous]00

On merit. Seriously, I this is a lot easier than this. It's just incentives. If you find handing out a reward has a desired result, you call that which you are rewarding a merit. It is a terminology for the target of an incentive application that proved to be working.

I think your problem is that you expect this to be somehow "just", that some level of cosmic justice to play a role in it.

Reply
[-][anonymous]00

Suppose you want your tribe to obey a set of rules. So you tell them the rules were made by an old man called God.

I am slightly annoyed by examples of theism always based on its most primitive form. Aquinas or Maimonides were leaps and bound more sophisticated than Bronze Age theism. Theism is a form of expression, like music, and Gregorian is not exactly like tribesmen beating drums.

Another relatedly annoying thing is that when people totally seriously debate about whether it is true or not or figure it is a big deal to realize it is not true. It is a form of expression, a genre. It's job is not being somewhere on the truth or untruth spectrum. Compare it to music. You can express the sentiments of war with military marches or Turisas, put people in that mood, or you can express the sentiments of peace by elevating kinds of classical music. The same way, you can tell people that god hath told you to smite the heathen, or that every man is a brother in christ. It is roughly the same way of arousing certain emotions. It does not have any business with being true or not.

I used music as an example, because then I can say with Max Weber that atheist is not the most perfect term of what I am, I don't find that a meaningful or useful term, because that suggests a truth/untruth approach, but like Weber I am "amusical to religion".

Reply
[-]Richard_Kennaway30

[telling people about god] is roughly the same way of arousing certain emotions. It does not have any business with being true or not.

That statement would have you executed for heresy in some former times. It is essential to most sects of the Christian faith (the Church of England being the major exception) that God really, truly, does exist, that Jesus walked the earth, was crucified, and resurrected, and so on.

Reply
[-][anonymous]10

I am bit confused now. Do you expect people to literally mean what they are saying, all the time? Obviously it is somethign you would not have said openly in former times, however, you could tacitly know it and act in this knowledge. Similarly, said statements are essential to the public image such churches project, but do you need to take it any more seriously than the corporate "mission statement" pages where everybody knows they are just PR and in practice their sole mission is to make money?

Reading LW sometimes I get the impression that being very interested in truth comes with the bias that maybe other people too are very interested in truth and really mean what they are saying. But of course it is not so. Most of what ever said publicly is PR and signalling. Related: a friend of mine has a theory that if you ever get the point you are elected Pope it is almost certain you are an atheist by then.

Here is an example. I am reading Edmund Burke's 1792 "Sketch of a Negro Code" which is an argument for the gradual abolition of slavery by first educating slaves and making them more fit for free life. It has actually sensible sounding ideas like letting everybody have a little hobby farm helps teaching self-sufficency values, that schooling is useful, that as long as people are not very educated perhaps controlling alcohol is good, and so on. And it also includes religious services and preaching as part of that education. Burke does not give half a damn what sect a church belongs to, which alone suggests he may have been at least partially agnostic: people who really think theism is literally true usually care about its "flavor". Burke basically considers churches something similar to schools. You got schools to learn facts and to churches to learn values, seems to be the idea there. And this is 1792. Already, a politician, who was considered kinda conservative and his nickname was "The Jesuit" did not give a damn about whether the "facts" the churches teach about Jesus are true or not. He was interested in the moral values and moral habits they teach.

So why are we still interested in the whole religion-as-truth-or-untruth? Because some almost illiterate people in the Bible Belt really seem to believe it? Just ignore them and focus on their politics, that is what matters for you. Because some very educated and smart people in the Vatican use it as a PR message? Ignore the message just like you would ignore Nike's corporate mission statement bullshit. Focus on what matters: churches as agents, institutions playing a role in society.

Reply
[-]Jiro50

Educated people in the Vatican wouldn't use factual claims about religion as a PR message unless the general populace believed that factual claims about religion are important, just like companies wouldn't have PR statements about responsibility if there weren't people reading those statements who actually thought it was important for the company to have responsibility.

Furthermore, I think you're committing typical mind fallacy. You're not realizing that some people can have thought processes that are alien to you. They can't possibly mean it, because you can't imagine yourself saying it and meaning it.

Reply
[-]Richard_Kennaway00

Obviously it is somethign you would not have said openly in former times, however, you could tacitly know it and act in this knowledge. Similarly, said statements are essential to the public image such churches project, but do you need to take it any more seriously than the corporate "mission statement" pages where everybody knows they are just PR and in practice their sole mission is to make money?

A fake can only exist if there is a reality for it to be a fake of. Gold exists and is valued, and so there is also fake gold. But mithril is a fictional metal, so there is no fake mithril, only pretend mithril. "God" only works as persuasion when people generally believe in God. "Let's pretend" may work for some groups of neopaganists, but I can't see society running on it.

Burke repudiated atheism and its close companion deism, which were well underway by 1792. He was not attached to Christianity in particular, but only because he believed some other religions to also have possession of divinely revealed truth. (I'm cribbing all this from here, btw.) Is his consistency in expressing these views to be taken as evidence that he believed the opposite?

And what Jiro said about typical mind fallacy.

Reply
[-]Quill_McGee00

I would disagree with the phrasing you use regarding 'human terminal values.' Now, I don't disagree that evolution optimized humans according to those criteria, but I am not evolution, and evolution's values are not my values. I would expect that only a tiny fraction of humans would say that evolution's values should be our values(I'd like to say 'none,' but radical neo-darwinians might exist). Now, if you were just saying that those are the values of the optimization process that produced humanity, I agree, but that was not what I interpreted you as saying.

Reply
[-]Furslid00

Savory and spirit are two different types of uncategories. Savory starts by having a well defined and narrow category, flavor. Then it uses negation to eliminate a portion of that category. The color green isn't sweet, but that doesn't make it savory because green isn't a flavor. I have some other valid uncategorical definitions of this type.

A mongrel is a dog that doesn't belong to any recognized breed of dog. Manslaughter is the killing of one human by another human, without the intent of seriously wounding or killing. Health is the state of a living organism without significant disease or injury. Fiction is any story that does not represent actual events.

Spirit is a problem because it is an unanchored uncategory. It doesn't start with a known subset of thingspace, instead it starts with thingspace. To use older terms, a definition is generally genus and differentia. Spirit lacks a genus in the way that savory doesn't.

Reply
[-]dxu00

What about a purely hedonistic utility function, i.e. a utility function that seeks to maximize the amount of subjective pleasure experienced by (if you're selfish) yourself or (if you're altruistic) everyone? This of course allows for utility monsters, but I'm not convinced that that's necessarily a bug as opposed to a feature, and it's the only solution I see to the "human values are ultimately arbitrary" problem.

Reply
[-]Jiro00

That leads to the repugnant conclusion.

Also, the way you describe it, it would lead to wireheading everyone.

Reply
[-]dxu00

That leads to the repugnant conclusion.

I am aware of this, and as my previous comment implies, I'm not convinced it's actually all that repugnant.

Also, the way you describe it, it would lead to wireheading everyone.

Possibly, yes. Presumably, if you don't value being wireheaded, you wouldn't like that (or more accurately, you would like that, but you wouldn't want that). But that just leads back to the question of what to value, which, as the article points out, is not at all straightforward to settle.

Reply
[-]Gunnar_Zarncke00

Generally an acceptably well written post. I don't exactly like your tone but I agree with the state facts to a large degree. I disagree most with this:

A human society's values are ultimately stated in terms of the gene alleles common in that society. These tend to be the values we think of as human values, because they often supervene on rationality and must be expressed explicitly. But, again, they benefit genes, not humans.

A human society's values are to a larger degree determined by how they benefit the society. One could argue to which degree this depends on the indirect benefit to the individual which indeed is "ultimately stated in terms of the gene alleles common in that society". But the causality is different and via a complex if not chaotic process. Societies compete among each other and a more cooperative one may out-compete a warrior society - or not.

All this is one more reason it is difficult to find a consistent stable volition.

Reply
[-]27chaos-40

Nothing new here.

Reply
[-]PhilGoetz40

It's good that nothing here seems new to you. But the final section implies that Friendly AI, as proposed by Eliezer, is impossible in principle if it relies on finding terminal human values, which don't exist. I'm sure there are people here who don't yet believe that.

FAI/CEV is supposed to find a set of values with reflective equilibrium, which is different than terminal values. But whether a reflective equilibrium exists, or is unique, or would encompass any values we didn't share with other mammals, is unclear.

Reply
[-]TrE00

But well-explained.

Reply
Moderation Log
More from PhilGoetz
View more
Curated and popular this week
68Comments