Can someone sketch me the Many-Worlds version of what happens in the delayed choice quantum eraser experiment? Does a last-minute choice to preserve or erase the which-path information affect which "worlds" decohere "away from" the experimenter? If so, how does that go, in broad outline? If not, what?

I wonder what probability epiphenomenalists assign to the theory that they are themselves conscious, if they admit that belief in consciousness isn't caused by the experiences that consciousness brings.

The more I think about it, the more absurdly self-defeating it sounds, and I have trouble believing that ANYONE could hold such views after having thought about it for a few minutes. The only reason I continue to think about it is because it's very easy to believe that some people, no matter how an AI acted and for how long, would never believe the AI to be conscious. And that bothers me a lot, if it affects their moral stance on that AI.

Wrote Something Story-like

Living in Weirdtopia: Week One

I see the pattern identity theory, where uploads make sense, as one that takes it as a starting point that you have an unambiguous past but no unambiguous future. You have moments of consciousness where you remember your past, which gives you identity, and lets you associate your past moments of consciousness to your current one. But there's no way, objective or subjective, to associate your present moment of consciousness to a specific future moment of consciousness, if there are multiple such moments, such as a high-fidelity upload and the original person, who remember the same past identity equally well. A continuity identity theorist thinks that a person who gets uploaded and then dies is dead. A pattern identity theorist thinks that people die in that sense several times a second and have just gotten used to it. There are physical processes that correspond to moments of consciousness, but there's no physical process for linking two consecutive moments of consciousness as the same consciousness, other than regular old long and short term memories.

There's no question that the upload and the original will diverge. If I have a non-destructive upload done on me, I expect to get up from the upload couch, not wake up in the matrix, old habits and all that. And there is going to be a future me who will experience exactly that. But if the upload was successful, there's also going to be a future me who will be very surprised to wake up staring at some fluorescent polygons, having expected to wake up on the upload coach. This is where the "no unambiguous future selves" stops being sophistry and starts paying rent for the pattern identity theorist. "Which one is the real me" is a meaningless question. All we have to go with are memories, and both of me will have my memories.

If you want to argue a pattern identity theorist out of it, you'll want to argue why there has to necessarily be more than just memory going on with producing the sense of moment-to-moment personal continuity, and why the physically unconnected moments of consciousness model can't be sufficient.

I was just taking that reasoning to the logical conclusion -- it applies just as well to the non productive elderly as it does to unneeded copies.

So then it would follow that when searching for a theory, the simplest ones will not always be the correct ones, since the observation-generating phenomenon was not chosen by nature to necessarily be the simplest phenomenon that could generate those observations. I think that may be what the essay is really getting at.

It might be a difference of starting points, then. We can either start with a universal approach, a broad prior, and use general heuristics like Occam's Razor, then move towards the specifics of a situation, or we can start with a narrow prior and a view informed by local context, to see how Nature typically operates in such domains according to the evidence of our intuitions, then try to zoom out. Of course both approaches have advantages in some cases, so what's actually being debated is their relative frequency.

I'm not sure of any good way to survey the problem space in an unbiased way to assess whether or not this assertion is typically true (maybe Monte Carlo simulations over random algorithms or something ridiculous like that?), but the point that adding unnecessary additional assumptions to a theory is flawed practice seems like a good heuristic argument suggesting we should generally assume simplicity. Does the fact that naive neural nets almost always fail when applied to out of sample data constitute a strong general argument against the anti-universalizing approach? Or am I just mixing metaphors recklessly here, with this whole "localism" thing? Simplicity and generalizability are more or less the same thing, right? Or is that question assuming the conclusion once again?

Did you look up the papers he referenced, then? Or are you speaking just based on your impression of his summaries? I too thought that his summaries were potentially misleading, but I failed to track down the papers he mentioned to verify that for certain.

I'm not sure if this is really an interesting thing to say, however - when talking about the actual phenomena, they are neither really simple nor complex. They have a single true explanation. It's only when humans are trying to establish the explanation based on limited observation that simplicity and complexity come into it.

This perspective is new to me. What are your thoughts on things like Salmonoff induction? It seems to me like that's sufficiently abstract that it requires simplicity is a meaningful idea even outside the human psyche. I cannot really imagine any thinking-like process that doesn't involve notions of simplicity.

Looking at the machine learning section of the essay, and the paper it mentions, I believe the author to be making a bit too strong a claim based on the data. When he says:

"In some cases the simpler hypotheses were not the best predictors of the out-of-sample data. This is evidence that on real world data series and formal models simplicity is not necessarily truth-indicative."

... he fails to take into account that many more of the complex hypotheses get high error rates than the simpler hypotheses (despite a few of the more complex hypotheses getting the smallest error rates in some cases), which still says that when you have a whole range of hypotheses, you're more likely to get higher error rates when choosing a single complex one than a single simple one. It sounds like he says Occam's Razor is not useful just because the simplest hypothesis isn't ALWAYS the most likely to be true.

Similarly, when he says:

"In a following study on artificial data generated by an ideal fixed 'answer', (Murphy 1995), it was found that a simplicity bias was useful, but only when the 'answer' was also simple. If the answer was complex a bias towards complexity aided the search."

This is not actually relevant to the discussion of whether simple answers are more likely to be fact than complex answers, for a given phenomenon. If you say "It turns out that you're more likely to be wrong with a simple hypothesis when the true answer is complex", this does not affect one way or the other the claim that simple answers may be more common than complex answers, and thus that simple hypotheses may be, all else being equal, more likely to be true than complex hypotheses when both match the observations.

That being said, I am sympathetic to the author's general argument. While complexity (elaboration), when humans are devising theories, tends to just mean more things which can be wrong when further observations are made, this does not necessarily point to whether natural phenomena is generally 'simple' or not. If you observe only a small (not perfectly representative) fraction of the phenomenon, then a simple hypothesis produced at this time is likely to be proven wrong in the end. I'm not sure if this is really an interesting thing to say, however - when talking about the actual phenomena, they are neither really simple nor complex. They have a single true explanation. It's only when humans are trying to establish the explanation based on limited observation that simplicity and complexity come into it.

If the many-worlds interpretation is truly how the world is, and if having multiple copies of myself as an upload is more valuable than just having one copy on more powerful (or distributed) hardware, then...

I could bid for a job asking for a price which could be adequate if I were working by myself. I could create N copies of myself to help complete the job. Then, assuming there's no easy way to meld my copies back into one, I could create a simple quantum lottery that deletes all but one copy.

Each copy is guaranteed to live on in its own Everett branch, able to enjoy the full reward from completing the job.

Firstly, thank you for creating this well-written and thoughtful post. I have a question, but I would like to start by summarising the article. My initial draft of the summary was too verbose for a comment, so I condensed it further - I hope I have still captured the main essense of the text, despite this rather extreme summarisation. Please let me know if I have misinterpreted anything.

People who predict doomsday scenarios are making one main assumption: that the AI will, once it reaches a conclusion or plan, EVEN if there is a measure of probability assigned to that conclusion or plan, act as if the conclusion is certain. That is, it will mindlessly carry out the action, even if its human instructors say it is the wrong action, even if the checking code in place to double check actions says that it's wrong (due to the instructors saying it's wrong).

You state that this all comes down to a 'Doctrine of Infallibility': if an AI cannot reassess or in some way take into account the uncertainty of its conclusions, then it might act as the doomsayers fear. But you also state that an AI containing such a Doctrine would be unable to be very intelligent, because it would contain a contradiction, a logical inconsistency: it would have both a full understanding of the uncertainty in its knowledge and reasoning methods, AND still be programmed to act as if it was certain of its conclusions. Such an AI would never reach human level AI, let alone be intelligent enough to be a threat to us.

Any assumption that an AGI would take our commands literally and instantly implement the naive (and catastrophic) actions to fulfil them is based on the assumption that the AI is sticking to the Doctrine of Infallibility, and is thus flawed.

Now, my question to you: do you think it would be possible in theory to create an AI which does not abide by the Doctrine of Infallibility (something like a Swarm Relaxation Intelligence), yet is STILL programmed to perform actions purely based on the consideration of all the types of uncertainty in its knowledge and reasoning methods? So for example, it considers all relevant facts, reaches a plan to fulfil its goal, continues to investigate relevant facts until its probability that this is the best plan reaches a certain threshold, and implements it? I know that in practice, we would have 'checking code' which makes it pass the plan by humans before doing it, in order for us to assess the sensibleness and safety of the plan. But in theory, would you consider it possible for the process to work without human input, maybe once the AI has reached a certain level of intelligence?