I would say that what I did is more like continuing to care about continuity, but trying to put it into causal continuity or pattern continuity after the particular hypothesis of 'particle identity continuity' turned out to be nonsense. Also, I regard this as a problem not strictly of utility functions because it controls what I expect to see happen after being cryonically revived or stepping into a Star Trek transporter - I either see the next moment, or see what happens after dying in a car crash i.e. NULL. Yes, I'm aware that this last part is confused, but just because I'm confused about something epistemic doesn't mean that it gets packed into utility functions. I shall have to write a post about this at some point.
I would say that what I did is more like continuing to care about continuity, but trying to put it into causal continuity or pattern continuity after the particular hypothesis of 'particle identity continuity' turned out to be nonsense.
It seems likely that we value several different kinds of continuity. I mentioned physical continuity (the kind that makes people reluctant to step into a destructive scanner, and which may reduce to something else even if not "particle identity continuity") and psychological continuity. (EDIT: Pattern continuity and causal continuity may be independent values or perhaps reductions for psychological continuity.) Are you assuming that we can only value one kind of continuity?
Also, I regard this as a problem not strictly of utility functions because it controls what I expect to see happen after being cryonically revived or stepping into a Star Trek transporter
What if we talk about other people instead of ourselves to isolate the problem to just utility functions?
What if we talk about other people instead of ourselves to isolate the problem to just utility functions?
My utility function is over 'what other people see happen to themselves' so it contains a reference to the same epistemic question.
My utility function is over 'what other people see happen to themselves' so it contains a reference to the same epistemic question.
Doesn't this depend on how your utility function defines "people"? If it's defined via pattern continuity, you get one answer to this question, and if it's defined via physical continuity (or perhaps a combination of physical and pattern continuity), you get another. (Much like how the "a tree falls in forest" question depends on how "sound" is defined.)
Note that if "people" were ontologically primitive, then there would be a single objective answer. People are not ontologically primitive in reality, but are in our usual models. So it seems reasonable that we might intuitively think there should be a single objective answer to "what will someone see when they step into a Star Trek transporter" when there really isn't.
If you don't mind a rather primitive question:
If I were to ask you 'which things are ontologically primitive in reality?', what kinds of things would you use to justify your answer? To be clear, I'm not just asking about what your answer is, but what kind of evidence you think is relevant to determining an answer. What, in other words, would things have to look like for you to conclude that human beings were ontologically primitive in reality (and not just in our usual models).
I ask, among other reasons, because although I'm confident that phenomena relevant to human beings, like behaviors, thoughts, biological processes, etc. are reducible to more fundamental physical systems, it's not obvious to me that this straightforwardly means that those more fundamental physical systems are more ontologically primitive than human beings. So far as I understand things, the physical, chemical, and biological theories we use to explain phenomena relevant to human beings don't purport to make claims about ontological primitiveness.
This topic probably deserves more thought than I've put into it, but it seems to me that you can tell what things are ontologically primitive in in reality by looking at what objects the fundamental laws of physics keep track of and directly operate upon. For example in Newtonian physics these would be individual particles, and in Quantum Mechanics it would just be the wavefunction. (Of course at this point we don't know what the fundamental laws of physics actually are so we can't say what things are ontologically primitive yet, but it seems pretty clear that it can't be human beings.)
it's not obvious to me that this straightforwardly means that those more fundamental physical systems are more ontologically primitive than human beings
Ontological primitiveness seems like a binary property. Either something is kept track of and operated upon directly by the fundamental laws of physics, or it isn't. I can't see what sense it would make to say one thing is "more primitive" than another.
(It may be that there is more than one concept of "ontological primitiveness" that is useful. I think my definition/explanation makes sense in combination with my recent posts and comments, but you may have another one in mind?)
it seems to me that you can tell what things are ontologically primitive in in reality by looking at what objects the fundamental laws of physics keep track of and directly operate upon.
Suppose some people constructed an AI which is programmed to experience the world in terms of ontologically primitive things from the get go, and construct the rest of its (non-primitive) ontology from there. Do you think an AI, experiencing only ontologically primitive things and their behaviors according to fundamental physical laws could discover the existence of, say, living things?
Do you think an AI, experiencing only ontologically primitive things and their behaviors according to fundamental physical laws could discover the existence of, say, living things?
What do you mean by "discover the existence of living things"? It seems plausible that such an AI may create some auxiliary (or "higher-level") objects in its world model to help it make predictions because it doesn't have enough computing power to just apply the fundamental laws of physics, and in the course of doing this may also label some such objects with a label that's roughly equivalent to "living". If this counts, I think the answer is yes, possibly, depending on the design of the AI.
It seems plausible that such an AI may create some auxiliary (or "higher-level") objects in its world model to help it make predictions because it doesn't have enough computing power to just apply the fundamental laws of physics
Assume it has infinite computing power. The AI thing is just a way of asking this question: if something knew all the facts about the things physical laws keep track of and directly operate on, and it were logically omniscient, would it know, for example, that this thing here is a tulip, that it's alive, etc.?
If not (I gather from your post that the answer is 'no') then it seems we should conclude one of two things:
1) Tulips are not in the territory, or,
2) Tulips are in the territory, but (for some reason) some facts about tulips are not derivable from facts about ontologically primitive things.
Which do you think is right? Or have I left out one or more possibilities?
(EDIT: I changed the example from 'me' to 'tulips' to avoid the impression that this question has anything to do with consciousness)
I'm also not sure what you mean by "Are tulips are in the territory?" or why you are asking me that. There seem to be collections or structures of ontologically primitive objects in the territory that correspond to the objects in our internal models that we label as "tulips". From this, can you derive for yourself whether "tulips are in the territory"?
I'm also not sure what you mean by "Are tulips are in the territory?" or why you are asking me that.
I'm trying to get some grip on the relation between ontologically primitive things and ontologically non-primitive things. A second question lurking about here is one raised by some of EY's recent talk about ontology as he would want it programmed into an AI.
We didn't all start by understanding ontological primitives and discover that human beings exist. We started with human beings and discovered that facts about human beings are reducible to facts about ontological primitives (discovering what those primitives were along the way). But does the fact that we went from human beings down to ontological primitives mean that something that started from ontological primitives would discover human beings?
But if the question isn't clear, or feels unmotivating, then I withdraw it, and I appreciate your answers thus far.
It's possible to detect tulips, but there are many alternative things that it's possible to detect, so there needs to be some motivation for the detecting of tulips in particular to actually take place. For natural concepts, it's efficient world modeling (which your AI by assumption doesn't need to care about), and for morality-related concepts, it's value judgments (these will require different concepts for different AIs, but may agree on the utility of keeping track of the "fundamental" physical facts).
(On a different note, "Are tulips in the territory?" sounds like a question about definitions. Some more specific relevant query may be similar, but I'm not sure how to find one.)
So you're saying that my AI (with infinite computational power) would never discover the existence of tulips?
(On a different note, "Are tulips in the territory?" sounds like a question about definitions.
I don't intend it to be. I think tulips exist, unlike shmulips (similar to tulips, except they have golf balls instead of flowers), which don't. I don't think I have a firm grip on the map-territory distinction, but I was trying to use it in the way Wei was using it.
Anyway, here's the basis of my question: tulips do exist. They're real, mind independent things and they are part of the furniture of the universe. Any god or AI who came into our universe would have an incomplete understanding of this universe if they failed to include tulips in their story.
That said, is the complete story of our universe derivable from a complete story of the ontological primitives (plus whatever logic you wish to avail yourself of)? I'm not totally sure that's a well formed question, mind you.
Anyway, here's the basis of my question: tulips do exist. They're real, mind independent things and they are part of the furniture of the universe. Any god or AI who came into our universe would have an incomplete understanding of this universe if they failed to include tulips in their story.
Tulips objectively exist as a fuzzy cluster in configuration space, and if an AI were to list all facts about the world, this would be one of them. But unlike us, an AI or god doesn't necessarily have a reason to notice this clustering or make any use of it. It's kind of like 22581959 being prime is an objective fact that you and I can discover, but don't necessarily have any reason to notice or make use of.
BTW, Eliezer argued, and I agree with, that this kind of objective clustering can't be used directly to define morally relevant concepts like "people".
Tulips objectively exist as a fuzzy cluster in configuration space, and if an AI were to list all facts about the world, this would be one of them.
I was being a bit ambiguous: I mean to talk about concrete individual tulips, not the species. Even given the framework in 'The Cluster Structure...", each actual tulip is a point in thingspace, not a cluster.
If an AI were to list all facts about the world, it would list that the wavefunction of the universe can be approximately factored into X, where X corresponds to what we would call an individual tulip. (Note that an individual tulip is also actually a cluster in configuration space, because it's a blob of amplitude-factor, not a single point in configuration space. Of course this cluster is much smaller than the cluster of all tulips.)
This topic probably deserves more thought than I've put into it, but it seems to me that you can tell what things are ontologically primitive in in reality by looking at what objects the fundamental laws of physics keep track of and directly operate upon. For example in Newtonian physics these would be individual particles, and in Quantum Mechanics it would just be the wavefunction.
The problem is that different equivalent formulations will make different things ontologically primitive.
(Of course at this point we don't know what the fundamental laws of physics actually are so we can't say what things are ontologically primitive yet, but it seems pretty clear that it can't be human beings.)
How do you know there is a fundamental level, as opposed something like a void cathedral?
The problem is that different equivalent formulations will make different things ontologically primitive.
Perhaps in this case we could say "the ontology of the universe is one or the other but I can't tell which, so I'll just have to be uncertain". Do you see any problems with this, or have any better ideas?
How do you know there is a fundamental level, as opposed something like a void cathedral?
Can you give an example of a mathematical formulation of a void cathedral, just to show that such a thing is possible?
Can you give an example of a mathematical formulation of a void cathedral, just to show that such a thing is possible?
One description is something like the following: take the space of computable universes that agree with our observations so far. Rather than putting an Occam prior over it, put an ultrafilter on it. One can pick the ultrafilter so that the set of universes where any particular level is fundamental has measure zero.
I'm afraid I lack the background knowledge and/or math skills to figure out your idea from this short description. I can't find any papers after doing a search either, so I guess this is your original idea? If so, why not write it up somewhere?
Do you really think we start out valuing physical continuity? I thought the opposite: the writers of the original Star Trek seemed to think nothing of including teleportation, nor does it cause alarm in D&D/Planescape or Harry Potter. I figured people might perhaps have a terminal value for causal continuity or connection, but the objection you point to stemmed from a false understanding of physics that made "physical continuity" important. (Compare someone with a little knowledge who decides to value 'unique' sets of human DNA, then later decides there aren't any.)
But my confidence decreased upon reflection. The Hebrew Bible with its lack of any clear 'soul' doctrine also lacks HP-style transformations of people (unless you count the implied snake with legs, or surgery producing Eve from a rib).
I thought the opposite: the writers of the original Star Trek seemed to think nothing of including teleportation, nor does it cause alarm in D&D/Planescape or Harry Potter.
I think the naive reaction to onscreen teleportation is that it is 'instantaneous movement' bypassing intervening distance, a la wormholes, not the destruction of an original and construction of a new version at a different location. If the Star Trek teleporter worked onscreen by slowly burning the original to ashes, and then growing a new copy at the destination in a vat, people would have very different reactions.
There was an episode of TNG about this. They wind up with two Rikers if I remember correctly. Then in an episode of Voyager two signals are accidentally combined resulting in a hybrid of two characters.
They wind up with two Rikers if I remember correctly.
OTOH, they all seem dreadfully puzzled at the issues of identity this raises. (It had happened some time ago, without anyone realizing; the original was left behind after they successfully beamed him up, and they only realized what had happened when they revisited the planet. So they had spent some time getting to know one copy, and the other was essentially a new character.)
You do realize the notion of the "clone" that somehow has the original's memories is a well-established pulp science fiction trope, and (when the original is dead) is usually treated as the same character?
I thought the opposite: the writers of the original Star Trek seemed to think nothing of including teleportation, nor does it cause alarm in D&D/Planescape or Harry Potter.
I'm not sure how this is evidence for your hypothesis 'the objection you point to stemmed from a false understanding of physics that made "physical continuity" important'. If you are right, wouldn't this false understanding of physics also apply to Star Trek (which is supposed to have the same physics as our world), and lead to the same objection? I think a more likely explanation is that for various reasons, the the details of how transporters are supposed to work and the implications thereof don't rise to salience in most viewers' minds.
In the case of D&D/Planescape and Harry Potter, I think the supposed physics/ontology of the fictional worlds are different enough from ours (they both have actual souls for example) and also sufficiently murky that we just say "if the in-world characters aren't worried about their teleportation spells, who am I to judge?"
I'm not sure how this is evidence for your hypothesis 'the objection you point to stemmed from a false understanding of physics that made "physical continuity" important'.
It's not, it's evidence that your "value" for physical continuity is not the same as your value for happiness or less pain. If Hairy is correct here, it would probably be a cached thought based on "if I'm destroyed, I will DIE. Teleportation destroys me. Therefore, if I teleport I will DIE."
If faced with the knowledge that physical continuity doesn't exist in the real world at the level of fundamental physics, one might conclude that it's crazy to continue to value it, and this is what Eliezer's post argued. But if we apply this reasoning in a non-selective fashion, wouldn't we also conclude that we should stop valuing things like "pain" and "happiness" which also do not seem to exist at the level of fundamental physics?
The relevant distinction isn't between whether something "exists at the level of fundamental physics" or not. The distinction is between whether something reduces to fundamental physics or not.
Perhaps some talk about macroscopic physical continuity can be reduced to fundamental physics. (What you say about widespread agreement on typical cases points to this.) But some talk about physical continuity makes no sense at all within the theoretical framework of quantum mechanics. It's not just that the physical continuity isn't fundamental. It's not just that it doesn't "have a simple definition in terms of fundamental physics". Rather, it can't be given any sense at all. That, at any rate, is the claim in posts like Can You Prove Two Particles Are Identical?. It makes more sense to say that a concept is morally irrelevant if it can't be referred to anything in reality at all.
Pain and happiness might reduce to fundamental physics, while physical continuity (in some cases) doesn't reduce to anything. There would then be no inconsistency in selectively discarding physical continuity but not pain and happiness.
It's not just that it doesn't "have a simple definition in terms of fundamental physics". Rather, it can't be given any sense at all.
The thing is, Eliezer jumped directly from "physical continuity can't be given sense in terms of persistent identity of fundamental particles" to "we should stop caring about physical continuity" while skipping the step of "physical continuity can't be given any other sense". How did he do this, except by relying implicitly on the assumption that it should "have a simple definition in terms of fundamental physics" or something similar, even as he would surely deny this assumption when considering it explicitly?
I would say that what I did is more like continuing to care about continuity, but trying to put it into causal continuity or pattern continuity after the particular hypothesis of 'particle identity continuity' turned out to be nonsense. Also, I regard this as a problem not strictly of utility functions because it controls what I expect to see happen after being cryonically revived or stepping into a Star Trek transporter - I either see the next moment, or see what happens after dying in a car crash i.e. NULL.
It seems like a decent attempt to deal with this sort of crisis is to think about what features in your new ontology cause you to label something as X in the old ontology.
For example, to recover values in your macroscopic physical continuity example...
For having mostly-the-same values:
Find the cognitive algorithms that cause you to label something as a macroscopic physical entity. Describe them in your new ontology in a way that lets you figure out if, within your new ontology, you would have labelled something as a distinct entity in your old ontology.
Probably more dangerous alternative:
You could try to figure out what about physical continuity it was that you valued, and what caused you to think that you valued it. If upon that reflection you decide to keep it, keep it.
This isn't a complete description of how to deal with it, nor is it like, mathematically formalized, but I'm curious as to what people think.
So basically: Redomain your utility function by composing it with an adaptor. Where the adaptor is a map from new-ontology -> old-ontology. Construct the adaptor by reverse-engineering your algorithms. Have I got that right?
Edit: No this sucks. Sometimes the old ontology doesn't make sense. I must think more. /Edit
That's a good statement of the problem, but I can see that "reverse engineer your algorithms" is the hard part, and we've just bottled it up as a black box. There's no obvious way to deal with cases that couldn't exist in your old ontology (brain damage can't exist in a simple dualist ontology, for example), or cases where there's a disagreement (teleportation and destructive-scan + print are different when things are ontologically basic, but more advanced physics says they are the same).
Some help may come from the fact that we seem to have some builtin support for ontology-shifting. It does happen successfully, though perhaps not always without loss. On the other hand people with the same ontology don't seem to diverge much by getting their through different update-chains.
Pretty accurate description of the mostly-the-same attempt. Also agreed that reverse-engineering your labelers is hard.
I think that for the other examples you would then need to do the more dangerous alternative and try and figure out what about your original concept you valued. It seems like you can do this with a mix of built-in hardware (if you're a human), and trying to come up with explanations about what would cause you to value something.
Like, for physical contiguity I value the fact that if I interact with a "person" then the result of that interaction will be causally relevant to them at some later time. That's very important if I want to like, interact with people in some way that I'll care about having done later. It would suck to make lunch plans with someone, and then have that be completely irrelevant to their later behavior/memory.
I also think that it's worth mentioning that I don't think that humans are always valuing things because they have a conceptual framework that implies they should. I'm not saying that ontologies change, but it doesn't feel like most updates restructure the concepts that something is expressed in. For instance I used to be pretty pro-abortion, but recently found out that, emotionally at least, I find it very upsetting for reasons mostly unrelated to my previous justifications.
I like your more dangerous alternative. There are many good occasions to re-think one's values, and this kind of ontological shift, especially where previously valued items seem to dissolve into murk, seems like one of them.
I agree with your main conclusion, though I would like to propose a different analogy: on which side of the road should one drive?
If faced with the knowledge that physical continuity doesn't exist in the real world at the level of fundamental physics, one might conclude that it's crazy to continue to value it
"The right side of the road" doesn't exist in the real world either at the level of fundamental physics; though that doesn't make it crazy to care about it. Things like social norms and contract and informal agreements are about well-defined concepts in the realm of everyday human experience.
Sure, there are some differences between things we are "built-in" to value, and explicit social norms, but if we can tolerate some "imprecision" for some, why not for the others?
I don't know about you, but I am neither willing nor able to change my terminal values based on knowledge of physics, only to extrapolate them better.
Karl recently reminded me of the post Timeless Identity by Eliezer Yudkowsky, which I noticed seems to be an instance of this.
For what it is worth I didn't see Eliezer discard any values in that post---either when he first posted it or when I reread it last week. He seems to have kept all his values. Everything 'added up to normal' at the end. He just ended up in a situation where he would be somewhat more competent at expressing those values in a reductionist description.
Of course the important question is not whether it makes sense at the fundamental level of physics. The important question is whether it influences the subjective experience of any being we care about (including ourselves).
Personally, I value not dying, subjectively. "Physical continuity" just seems to be an obvious requirement, and that is why most reasonable people would not step into a cut & paste transporter. But really, it's safe (even if my guts don't seem to trust my brain right now).
On the other hand I value very much being happy and not in pain, for the two describe quite different subjective experiences.
As I mentioned in the previous post, our values seem to be defined in terms of a world model where people exist as ontologically primitive entities ruled heuristically by (mostly intuitive understandings of) physics and psychology. In this kind of decision system, both identity-as-physical-continuity and identity-as-psychological-continuity make perfect sense as possible values, and it seems humans do "natively" have both values. A typical human being is both reluctant to step into a teleporter that works by destructive scanning, and unwilling to let their physical structure be continuously modified into a psychologically very different being.
If faced with the knowledge that physical continuity doesn't exist in the real world at the level of fundamental physics, one might conclude that it's crazy to continue to value it, and this is what Eliezer's post argued. But if we apply this reasoning in a non-selective fashion, wouldn't we also conclude that we should stop valuing things like "pain" and "happiness" which also do not seem to exist at the level of fundamental physics?
Physical continuity doesn't really have any benefits that enter in at the level of our consciousness though. The fact that we don't have pain on a particulate level doesn't affect the sensation that you'll feel if you grab a hot iron, whereas if the atoms of your hand are gradually replaced with other atoms, you're not going to be gripped with the visceral sensation that it's not your hand, and have a need to draw on some philosophical argument to try to disabuse yourself of that notion.
"You can't reject absolutes without un-restraining certain particulars -that should remain just that- to replace it" is this a fair description of your position here Wei?
If faced with the knowledge that physical continuity doesn't exist in the real world at the level of fundamental physics, one might conclude that it's crazy to continue to value it, and this is what Eliezer's post argued. But if we apply this reasoning in a non-selective fashion, wouldn't we also conclude that we should stop valuing things like "pain" and "happiness" which also do not seem to exist at the level of fundamental physics?
Whoa, hold on there. Eliezer's post argued that what what we do care about doesn't correspond to physical continuity. It doesn't say anything about any other correspondences for other abstractions like pain and happiness.
I found my distaste for teleporting either never existed or was lost at a very young age, I think due to Star Trek teaching me that I ought to anticipate reappearing. Would people raised in an environment where physical continuity was not valued be persuadable that they should? I doubt it, but then there would be biases in play there too ...
My impression is that you're atypical in this. I watched Star Trek, read books with teleporting, and so on, and while I was completely unbothered by its portrayal in fiction (mostly because, I think, they deliberately showed it as non-destructive), I nonetheless have a visceral aversion to the prospect of personally being teleported by destructive means. (The issue is also, for example, explicitly discussed in Dan Simmons' Ilium and most of the characters are suitably horrified when they discover that the teleporters they've been using are destructive, although they get over it in the interests of expediency.)
In a previous post, I argued that nihilism is often short changed around here. However I'm far from certain that it is correct, and in the mean time I think we should be careful not to discard our values one at a time by engaging in "selective nihilism" when faced with an ontological crisis, without even realizing that's what's happening. Karl recently reminded me of the post Timeless Identity by Eliezer Yudkowsky, which I noticed seems to be an instance of this.
As I mentioned in the previous post, our values seem to be defined in terms of a world model where people exist as ontologically primitive entities ruled heuristically by (mostly intuitive understandings of) physics and psychology. In this kind of decision system, both identity-as-physical-continuity and identity-as-psychological-continuity make perfect sense as possible values, and it seems humans do "natively" have both values. A typical human being is both reluctant to step into a teleporter that works by destructive scanning, and unwilling to let their physical structure be continuously modified into a psychologically very different being.
If faced with the knowledge that physical continuity doesn't exist in the real world at the level of fundamental physics, one might conclude that it's crazy to continue to value it, and this is what Eliezer's post argued. But if we apply this reasoning in a non-selective fashion, wouldn't we also conclude that we should stop valuing things like "pain" and "happiness" which also do not seem to exist at the level of fundamental physics?
In our current environment, there is widespread agreement among humans as to which macroscopic objects at time t+1 are physical continuations of which macroscopic objects existing at time t. We may not fully understand what exactly it is we're doing when judging such physical continuity, and the agreement tends to break down when we start talking about more exotic situations, and if/when we do fully understand our criteria for judging physical continuity it's unlikely to have a simple definition in terms of fundamental physics, but all of this is true for "pain" and "happiness" as well.
I suggest we keep all of our (potential/apparent) values intact until we have a better handle on how we're supposed to deal with ontological crises in general. If we convince ourselves that we should discard some value, and that turns out to be wrong, the error may be unrecoverable once we've lived with it long enough.