I recently read this blog post by Allen Downey in response to a reddit post in response to Julia Galef's video about the Sleeping Beauty problem. Downey's resolution boils down to a conjecture that optimal bets on lotteries should be based on one's expected state of prior information just before the bet's resolution, as opposed to one's state of prior information at the time the bet is made.

I suspect that these two distributions are always identical. In fact, I think I remember reading in one of Jaynes' papers about a requirement that any prior be invariant under the acquisition of new information. That is to say, the prior should be the weighted average of possible posteriors, where the weights are the likelihood that each posterior would be acheived after some measurement. But now I can't find this reference anywhere, and I'm starting to doubt that I understood it correctly when I read it.

So I have two questions:

1) Is there such a thing as this invariance requirement? Does anyone have a reference? It seems intuitive that the prior should be equivalent to the weighted average of posteriors, since it must contain all of our prior knowledge about a system. What is this property actually called?

2) If it exists, is it a corollary that our prior distribution must remain unchanged unless we acquire new information?

Just came across this interview with Robert Aumann. On pgs. 20-27 he describes why and how he believes in Orthodox Judaism. I don't really understand what he's saying. Key quote (I think):

H (interviewer): Take for example the six days of creation; whether or not this is how it happened is practically irrelevant to one is decisions and way of conduct. It is on a different level.

A (Aumann): It is a different view of the world, a different way of looking at the world. That is why I prefaced my answer to your question with the story about the roundness of the world being one way of viewing the world. An evolutionary geological perspective is one way of viewing the world. A different way is with the six days of creation. Truth is in our minds. If we are sufficiently broad-minded, then we can simultaneously entertain different ideas of truth, different models, different views of the world.

H: I think a scientist will have no problem with that. Would a religious person have problems with what you just said?

A: Different religious people have different viewpoints. Some of them might have problems with it. By the way, I'm not so sure that no scientist would have a problem with it. Some scientists are very doctrinaire.

Anybody have a clue what he means by all this? Do you think this is a valid way of looking at the world and/or religion? If not, how confident are you in your assertion? If you are very confident, on what basis do you think you have greatly out-thought Robert Aumann?

Please read the source (all 7 pages I referenced, rather than just the above quote), and think about it carefully before you answer. Robert Aumann is an absolutely brilliant man, a confirmed Bayesian, author of Aumann's Agreement Theorem, Nobel Prize winner, and founder / head of Hebrew University's Center for the Study of Rationality. Please don't strawman his arguments or simply dismiss them!

Personal Statement

I like to think about big questions from time to time. A fancy that quite possibly causes me more harm than good. Every once in a while I come up with some idea and wonder "hey, this seems pretty good, I wonder if anyone is taking it seriously?" Usually, answering that results at worst in me wasting a couple days on google and blowing $50 on amazon before I find someone who’s going down the same path and can tell myself. "Well, someone's got that covered". This particular idea is a little more stubborn and the amazon bill is starting to get a little heavy. So I cobbled together this “paper” to get this idea out there and see where it goes.  

I've been quite selective here and have only submitted it on two other places Vixra, and FXQI forum.  Vixra for posterity in the bizarre case that it's actually right.  FXQI because they play with some similar ideas (but the forum turned out to be not really vibrant for such things).  I'm now posting it on Less Wrong because you guys seem to be the right balance of badass skeptics and open minded geeks.  In addition I see a lot of cool work on Anthropic Reasoning and the like so it seems to go along with your theme.

Any and all feedback is welcome, I'm a good sport!

Abstract

A popular objection to the Many-worlds interpretation of Quantum Mechanics is that it allows for quantum suicide where an experimenter creates a device that instantly kills him or leaves him be depending the output of a quantum measurement, since he has no experience of the device killing him he experiences quantum immortality. This is considered counter-intuitive and absurd. Presented here is a speculative argument that accepts counter-intuitiveness and proposes it as a new approach to physical theory without accepting some of the absurd conclusions of the thought experiment. The approach is based on the idea that the Universe is Fragile in that only a fraction of the time evolved versions retain the familiar structures of people and planets, but the fractions that do not occur are not observed. This presents to us as a skewed view of physics and only by accounting for this fact (which I propose calling the Continual Anthropic Principle) can we understand the true fundamental laws.

Preliminary reasoning

Will a supercollider destroy the Earth?

A fringe objection to the latest generation of high energy supercolliders was they might trigger some quantum event that would destroy the earth such as by turning it to strangelets (merely an example). To assuage those fears it has been noted that since Cosmic Rays have been observed with higher energies then the collisions these supercolliders produce that if a supercollider were able to create such Earth-destroying events cosmic rays would have already destroyed the Earth. Since that hasn't happened physics must not work that way and we thus must be safe.

A false application of the anthropic principle

One may try to cite the anthropic principle as an appeal against the conclusion that physics disallows Earth-destruction by said mechanism. If the Earth were converted to strangelets, there would be no observers on it. If the right sort of multiverse exists, some Earths will be lucky enough to escape this mode of destruction. Thus physics may still allow for strangelet destruction and supercolliders may still destroy the world. We can reject that objection by noting that if that were the case, it is far more probable that our planet would be alone in a sea of strangelet balls that were already converted by highenergy cosmic rays. Since we observe other worlds made of ordinary matter, we can be sure physics doesn't allow for the Earth to be converted into strange matter by interactions at Earth’s energy level.

Will a supercollider destroy the universe?

Among the ideas on how supercolliders will destroy the world there are some that destroy not just the Earth but entire universe as well. A proposed mechanism is in triggering vacuum energy to collapse to a new lower energy state. By that mechanism the destructive event spreads out from the nucleation site at the speed of light and shreds the universe to something completely unrecognizable. In the same way cosmic rays rule out an Earth-destroying event it has said that this rules out a universe destroying event.

Quantum immortality and suicide

Quantum suicide is a thought experiment there is a device that measures a random quantum event, and kills an experimenter instantly upon one outcome, and leaves him alive upon the other. If Everett multiple worlds is true, then no matter how matter how many times an experiment is performed, the experimenter will only experience the outcome where he is not killed thus experiencing subjective immortality. There are some pretty nutty ideas about the quantum suicide and immortality, and this has been used as an argument against many-worlds. I find the idea of finding oneself for example perpetually avoiding fatal accidents or living naturally well beyond any reasonable time to be mistaken (see objections). I do however think that Max Tegmark came up with a good system of rules on his "crazy" page for how it might work: http://space.mit.edu/home/tegmark/crazy.html

The rules he outlines are: "I think a successful quantum suicide experiment needs to satisfy three criteria:

1. The random number generator must be quantum, not classical (deterministic), so that you really enter a superposition of dead and alive.

2. It must kill you (at least make you unconscious) on a timescale shorter than that on which you can become aware of the outcome of the quantum coin-toss - otherwise you'll have a very unhappy version of yourself for a second or more who knows he's about to die for sure, and the whole effect gets spoiled.

3. It must be virtually certain to really kill you, not just injure you.”

Have supercolliders destroyed the universe? 

Let's say that given experiment has a certain "probability" (by a probabilistic interpretation of QM) of producing said universe destructive event. This satisfies all 3 of Tegmark's conditions for a successful quantum suicide experiment. As such the experimenter might conclude that said event cannot happen. However, he would be mistaken, and a corresponding percentage of successor states would in fact be ones where the event occurred. If the rules of physics are such that an event is allowed then we have a fundamentally skewed perceptions of what physics are.

It's not a bug it's a feature!

If we presume such events could occur, we have no idea how frequent they are. There's no necessary reason why they need to be confined to rare high energy experiments and cosmic rays. Perhaps it dictates more basic and fundamental interactions. For instance certain events within an ordinary atomic nucleus could create a universe-destroying event. Even if these events occur at an astonishing rate, so long as there's a situation where the event doesn't occur (or is "undone" before the runaway effect can occur), it would not be contradictory with our observation. The presumption that these events don't occur may be preventing us from understanding a simpler law that describes physics in a certain situation in favor of more complex theories that limit behavior to that which we can observe.

Fragile Universe Hypothesis

Introduction

Because of this preliminary reasoning I am postulating what I call the "Fragile Universe Hypothesis". The core idea is that our universe is constantly being annihilated by various runaway events initiated by quantum phenomena. However, because for any such event there's always a possible path where such event does not occur, and since all possible paths are realized we are presented with an illusion of stability. What we see as persistent structures in the universe (chairs, planets, galaxies) are so only because events that destroy them by and large destroy us as well. What we may think are fundamental laws of our universe, are merely descriptions of the nature of possible futures consistent with our continued existence.

Core theory

The hypothesis can be summarized as postulating the following:

1. For a given event at Time T there are multiple largely non-interacting future successor events at T + ε (i.e. Everett Many Worlds is either correct or at least on the right track)

2. There are some events where some (but not all) successor events trigger runaway interactions that destroy the universe as we know it. Such events expand from the origin at C and immediately disrupt the consciousness of any being it encounters.

3. We experience only a subset of possible futures and thus have a skewed perspective of the laws of physics.

4. To describe the outcome of an experiment we must first calculate possible outcomes then filter out those that result in observer destruction (call it the "continual anthropic principle")

Possible Objections

"If I get destroyed I die and will no longer have experiences. This is at face value absurd"

I'm sympathetic, and I'd say this requires a stretch of imagination to consider. But do note that under this hypothesis, no one will ever have an experience that isn't followed by a successive experience (see quantum immortality for discussion of death). So from our perspective our existence will go on unimpeded. As an example, consider a video game save. The game file can be saved, copied, compressed, decompressed, moved from medium to medium (with some files being deleted after being copied to a new location). We say that the game continues so long as someone plays at least one copy of the file. Likewise for us, we say life (or the universe as we know it) goes on so long as at least one successor continues.

"This sort of reasoning would result in having to accept absurdities like quantum immortality"

I don't think so. Quantum immortality (the idea that many worlds guarantees one immortality as there will always be some future state in which one continues to exist) presumes that personhood is an all-ornothing thing. In reality a person is more of a fragmented collection of mental processes. We don't suddenly stop having experiences as we die, rather the fragments unbind, some live on in the memory of others or in those experiencing the products of our expression, while others fade out. A destructive event of the kind proposed would absolutely be an all-or-nothing affair. Either everything goes, or nothing goes.

"This isn't science. What testable predictions are you making? Heck you don't even have a solid theory" 

Point taken! This is, at this point, speculation, but I think at this point it might have the sort of elegance that good theories have. The questions that I have are:

1. Has this ever been seriously considered? (I’ve done some homework but undoubtedly not enough).

2. Are there any conceptual defeaters that make this a nonstarter?

3. Could some theories be made simpler by postulating a fragile universe and continual anthropic principle?

4. Could those hypothetical theories make testable predictions?

5. Have those tests been consistent with the theory.

My objective in writing this is to provide an argument against 2, and starting to look into 1 and 3. 4 and 5 are essential to good science as well too, but we’re simply not at that point yet.

Final Thoughts

The Copernican Principle for Many worlds

When we moved the Earth as the center of the solar system, the orbits of the other planets became simpler and clearer. Perhaps physical law can be made simpler and clearer when we move the futures we will experience away from the center of possible futures. And like the solar system's habitable zone, perhaps only a small portion of futures are habitable.

Why confine the Anthropic Principle to the past? 

Current models of cosmology limit the impact of the Anthropic selection on the cosmos to the past: string landscapes, bubble universes or cosmic branes, these things all got fixed at some set of values 13 billion years ago and the selection effect does no more work at the cosmic scale. Perhaps the selection effect is more fundamental then that. Could it be that instead 13 billion years ago is when the anthropic selection merely switched from being creative in sowing our cosmic seeds to conservative in allowing them to grow? 

Part 2 – Winstrumental

 Part 1 is here.

The forgotten fifth virtue

Remember, you can't be wrong unless you take a position. Don't fall into that trap.

-- Scott Adams, Dogbert's Top Secret Management Handbook

CronoDAS posted this in a reply to my poem, and I dismissed him because my typical mind is typical. I would never make that mistake, so I didn’t think it’s a big deal. But it is. In the comments to part 1 a lot of people are heartily disagreeing with everything I wrote. I admire and respect them.  I already made a correction to a part of the post which was wrong. Unfortunately, a lot of people reading this couldn’t disagree if they wanted to, because they don’t have an account. I get that lurking is fun, but if you’re spending hours and hours on LessWrong and not posting anything I think you’re doing yourself a disservice.

In part 1 I speculated a lot about what goes on in Eliezer’s mind, knowing full well that Eliezer could read this and say that I’m wrong and I will have no comeback but pure embarrassment. What kind of foolhardy dunce would risk such a thing? Let me answer with another question: how else could I possibly change my mind? After reading them for a year, I have strong opinions on the goals and lessons of the sequences, and the only way to find out if I’m right or wrong is to open myself up to challenge. Worst case: people agree with me and I get sweet sweet karma. Best case: I become wiser. Am I at risk of sticking to an opinion too long just because I wrote it down? Yes, but I know I have that bias, anything known is something I can adjust for. If I don’t argue I don’t know what I don’t know.

If you want a chance to change your opinions, you have to put them where they can hurt you.  Or to use an Umeshism:  if you’ve never been proven an idiot on the internet you’re not learning enough from the internet.

Back to Harvard

Why don’t the psychologists at Harvard switch to reviewing nameless CVs? Well, why would they? They are tenured Harvard professors, they already won!  There was no bias shown for assessing stellar CVs, only those on the margins. So they’re not missing out on any superstars, at worst they hire some gentleman who would be their 32^nd strongest faculty member instead of a lady who would be 29^th. Would you cause a fuss if you were there?

In “Thinking Fast and Slow” Kahnemann writes that he noticed suffering from the halo effect when grading student exams. If a student did well on herfirst essay Kahnemann gave her the benefit of the doubt on later questions. He switched to grading all the answers to question 1, then all the answers to question 2 and so on. It took more time, but the grades were more accurate and fair. What’s my point? I guess it’s possible to “win at rationality” without a strong incentive, just maybe it takes a Nobel-level rationalist to do so.

Winning isn’t everything?

Vince Lombardi said that “Winning isn’t everything, it’s the only thing.” Aren’t you jealous of him? It’s so simple! I think the most common question asked of our community, mostly by our community, is why we don’t “win” as much as we think we are supposed to. In a rare display of good sense, I’m not going to speculate about why any of you don’t win, I’ll talk about myself.

My job isn’t as interesting, meaningful and full of potential as I would hope for. Why don’t I apply rationality to win at building a better career? Because when I think about it I remember that my job is also decently paying, secure, and full of decent people. My job is easy, and winning is hard. When I read about Nate Soares trying to save the world I feel a little inspired and a little ashamed that I’m not. Nate is almost certainly a better mathematician that I am, but I don’t think there’s a gargantuan gap between us. The big gap between Nate and me is in the desire to win. In my heart of hearts, I just don’t want to save the world as much as he does.

Love wins

What could I possibly want more than saving the world?

There are two ladies, let’s call them Rachel and Leah since my username is reminiscent of the Biblical Jacob. I met Rachel at the desert well (OKCupid) and we went on a few dates and at the same time Leah also replied to me on OKCupid and we also went on a few dates.  Then there were some situations and complications and my desire not to be an asshole so I decided that I had to choose one. The basic heuristic I would normally use pointed slightly to Rachel, but I kept vacillating back and forth for a few days, they were both much more attractive than any other girl I ever met through the site. Suddenly it hit me like Chuck Norris: this is an important decision, with huge stakes, one that I would have to make based on incomplete information with my brain biology trying to trip me up every step of the way. Might not this call for some EWOR?

I got to work. I introspected on past relationships and read the relevant science literature to come up with a weighted list of qualities I am looking for to maximize my chances of a happy long-term relationship. I wrote down all the evidence that could affect my assessment each quality for each lady, and employed every method I could think of to debias myself and give my best guess at the ratings. Then I peeked for the first time at the final score, and it was very surprising. My gut expected Rachel to be slightly ahead, but Leah won handily. I stared at the numbers for a while. Maybe I was too critical here? Overweighted this category there? No! The ghost of Eliezer wouldn’t let me change the bottom line from a formula to a value cell. And then, after 30 minutes of staring at the numbers, my intuition started catching up. For example, my impression from the first date was that Leah wasn’t very funny, and it stuck. When I actually wrote down the evidence, I remembered that she cracked me up once on our second date and a couple of times on our third date as she was slowly beginning to open up and trust me. I gave her a higher rating on humor-compatibility than I thought I would. I closed the spreadsheet and went to sleep. Two days later I broke up with Rachel.

Was I accurate in assessing Leah? Not exactly. She’s above and beyond anything I could’ve guessed. If I don’t “win” a single thing more from my rationality training than the few months I have gotten to spend so far with her, I’ve won enough.

Did I just praise disagreement?

I told this story about Leah to someone at a rationalist gathering. I thought he might  congratulate me on my achievement in rationality or denounce me as a cold and heartless robot. His actual reaction caught me completely by surprise: he just flat out didn’t believe me. He said that I probably used a spreadsheet to justify after the fact a decision that my gut had already made. The idea of someone applying something like EWOR that belongs on internet forums, to something like picking a woman to date was so foreign to him that he rejected it outright. I could almost hear him screaming separate magisteria!

Getting to the points

I’m no good at writing pithy summaries.  If you saw a good point anywhere in those two posts, grab it. I can’t help you. For what it’s worth, here’s Jacobian’s guide to actually using rationality to win:

1.     If you don’t believe you can, Luke, don’t bother. But if you’re not sure whether it works, wouldn’t it be interesting to find out?

2.     Taking ideas seriously requires work, maybe even *gasp* doing math. If you disagree with Eliezer or anyone else on a matter of math or science, sit down and figure it out. Don’t just read stuff, write stuff. Write a bit of code that simulates a probability problem. Derive something from Shrodinger’s equation on a piece of paper. Reading stuff is useful, but it’s not work; rationality is work.

3.     If there’s an opinion that you’re afraid you may be irrationally attached to and you have a real desire to find out the truth, post it on LessWrong. Don’t post things that are 99.999% true, they probably are. Post what you’re 80% sure about, that’s a 20% chance to really learn something. People will call you an idiot online, that’s what the internet is for. Losing karma is how you become smarter, it’s quite a thrill.

4.     Rationality will not change your entire life at once. Pick one thing that you want to win at and apply rationality to it. Just one, but one where you’ll know if you won or lost, so “being wiser” doesn’t count. Getting laid counts. If you take an L, you’ll learn a lot. If you win, you’ll know that the force is yours to command.

Who knows, maybe in a few years you’ll think you’re strong enough to save the world or something.

I'm sure that many of you here have read Quantum Computing Since Democritus. In the chapter on the anthropic principle the author presents the Dice Room scenario as a metaphor for human extinction. The Dice Room scenario is this:

1. You are in a world with a very, very large population (potentially unbounded.)

2. There is a madman who kidnaps 10 people and puts them in a room.

3. The madman rolls two dice. If they come up snake eyes (both ones) then he murders everyone.

4. Otherwise he releases everyone, then goes out and kidnaps 10 times as many people as before, and returns to step 3. 

The question is this: if you are one of the people kidnapped at some point, what is your probability of dying? Assume you don't know how many rounds of kidnappings have preceded yours.

As a metaphor for human extinction, think of the population of this world as being all humans who ever have or ever may live, each batch of kidnap victims as a generation of humanity, and rolling snake eyes as an extinction event.

The book gives two arguments, which are both purported to be examples of Bayesian reasoning:

1. The "proximate risk" argument says that your probability of dying is just the prior probability that the madman rolls snake eyes for your batch of kidnap victims -- 1/36.

2. The "proportion murdered" argument says that about 9/10 of all people who ever go into the Dice Room die, so your probability of dying is about 9/10.

Obviously this is a problem. Different decompositions of a problem should give the same answer, as long as they're based on the same information.

I claim that the "proportion murdered" argument is wrong. Here's why. Let pi(t) be the prior probability that you are in batch t of kidnap victims. The proportion murdered argument relies on the property that pi(t) increases exponentially with t: pi(t+1) = 10 * pi(t). If the madman murders at step t, then your probability of being in batch t is

  pi(t) / SUM(u: 1 <= u <= t: pi(u))

and, if pi(u+1) = 10 * pi(u) for all u < t, then this does indeed work out to about 9/10. But the values pi(t) must sum to 1; thus they cannot increase indefinitely, and in fact it must be that pi(t) -> 0 as t -> infinity. This is where the "proportion murdered" argument falls apart.

For a more detailed analysis, take a look at

http://bayesium.com/doomsday-and-the-dice-room-murders/

This forum has a lot of very smart people who would be well-qualified to comment on that analysis, and I would appreciate hearing your opinions.

A well-known American federal appellate judge, Alex Kozinski, has written a commentary on systemic biases and institutional myths in the criminal justice system.

The basic thrust of his criticism will be familiar to readers of the sequences and rationalists generally. Lots about cognitive biases (but some specific criticisms of fingerprints and DNA evidence as well). Still, it's interesting that a prominent federal judge -- the youngest when appointed, and later chief of the Ninth Circuit -- would treat some sacred cows of the judiciary so ruthlessly. 

This is specifically a criticism of U.S. criminal justice, but, ceteris paribus, much of it applies not only to other areas of U.S. law, but to legal practices throughout the world as well.

Following some somewhat misleading articles quoting me, I thought I’d present the top 9 myths about the AI risk thesis:

1. That we’re certain AI will doom us. Certainly not. It’s very hard to be certain of anything involving a technology that doesn’t exist; we’re just claiming that the probability of AI going bad isn’t low enough that we can ignore it.
2. That humanity will survive, because we’ve always survived before. Many groups of humans haven’t survived contact with more powerful intelligent agents. In the past, those agents were other humans; but they need not be. The universe does not owe us a destiny. In the future, something will survive; it need not be us.
3. That uncertainty means that you’re safe. If you’re claiming that AI is impossible, or that it will take countless decades, or that it’ll be safe... you’re not being uncertain, you’re being extremely specific about the future. “No AI risk” is certain; “Possible AI risk” is where we stand.
4. That Terminator robots will be involved. Please? The threat from AI comes from its potential intelligence, not from its ability to clank around slowly with an Austrian accent.
5. That we’re assuming the AI is too dumb to know what we’re asking it. No. A powerful AI will know what we meant to program it to do. But why should it care? And if we could figure out how to program “care about what we meant to ask”, well, then we’d have safe AI.
6. That there’s one simple trick that can solve the whole problem. Many people have proposed that one trick. Some of them could even help (see Holden’s tool AI idea). None of them reduce the risk enough to relax – and many of the tricks contradict each other (you can’t design an AI that’s both a tool and socialising with humans!).
7. That we want to stop AI research. We don’t. Current AI research is very far from the risky areas and abilities. And it’s risk aware AI researchers that are most likely to figure out how to make safe AI.
8. That AIs will be more intelligent than us, hence more moral. It’s pretty clear than in humans, high intelligence is no guarantee of morality. Are you really willing to bet the whole future of humanity on the idea that AIs might be different? That in the billions of possible minds out there, there is none that is both dangerous and very intelligent?
9. That science fiction or spiritual ideas are useful ways of understanding AI risk. Science fiction and spirituality are full of human concepts, created by humans, for humans, to communicate human ideas. They need not apply to AI at all, as these could be minds far removed from human concepts, possibly without a body, possibly with no emotions or consciousness, possibly with many new emotions and a different type of consciousness, etc... Anthropomorphising the AIs could lead us completely astray.

1. That AIs have to be evil to be dangerous. The majority of the risk comes from indifferent or partially nice AIs. Those that have some goal to follow, with humanity and its desires just getting in the way – using resources, trying to oppose it, or just not being perfectly efficient for its goal.
2. That we believe AI is coming soon. It might; it might not. Even if AI is known to be in the distant future (which isn't known, currently), some of the groundwork is worth laying now.

I am currently learning about the basics of decision theory, most of which is common knowledge on LW. I have a question, related to why EDT is said not to work.

Consider the following Newcomblike problem: A study shows that most people who two-box in Newcomblike problems as the following have a certain gene (and one-boxers don't have the gene). Now, Omega could put you into something like Newcomb's original problem, but instead of having run a simulation of you, Omega has only looked at your DNA: If you don't have the "two-boxing gene", Omega puts $1M into box B, otherwise box B is empty. And there is $1K in box A, as usual. Would you one-box (take only box B) or two-box (take box A and B)? Here's a causal diagram for the problem:

Since Omega does not do much other than translating your genes into money under a box, it does not seem to hurt to leave it out:

I presume that most LWers would one-box. (And as I understand it, not only CDT but also TDT would two-box, am I wrong?)

Now, how does this problem differ from the smoking lesion or Yudkowsky's (2010, p.67) chewing gum problem? Chewing Gum (or smoking) seems to be like taking box A to get at least/additional $1K, the two-boxing gene is like the CGTA gene, the illness itself (the abscess or lung cancer) is like not having $1M in box B. Here's another causal diagram, this time for the chewing gum problem:

As far as I can tell, the difference between the two problems is some additional, unstated intuition in the classic medical Newcomb problems. Maybe, the additional assumption is that the actual evidence lies in the "tickle", or that knowing and thinking about the study results causes some complications. In EDT terms: The intuition is that neither smoking nor chewing gum gives the agent additional information.

Summary: Utilitarianism is often ill-defined by supporters and critics alike, preference utilitarianism even more so. I briefly examine some of the axes of utilitarianism common to all popular forms, then look at some axes unique but essential to preference utilitarianism, which seem to have received little to no discussion – at least not this side of a paywall. This way I hope to clarify future discussions between hedonistic and preference utilitarians and perhaps to clarify things for their critics too, though I’m aiming the discussion primarily at utilitarians and utilitarian-sympathisers.

http://valence-utilitarianism.com/?p=8

I like this essay particularly for the way it breaks down different forms of utilitarianism to various axes, which have rarely been discussed on LW much.

For utilitarianism in general:

Many of these axes are well discussed, pertinent to almost any form of utilitarianism, and at least reasonably well understood, and I don’t propose to discuss them here beyond highlighting their salience. These include but probably aren’t restricted to the following:

• What is utility? (for the sake of easy reference, I’ll give each axis a simple title – for this, the utility axis); eg happiness, fulfilled preferences, beauty, information(PDF)
• How drastically are we trying to adjust it?, aka what if any is the criterion for ‘right’ness? (sufficiency axis); eg satisficing, maximising[2], scalar
• How do we balance tradeoffs between positive and negative utility? (weighting axis); eg, negative, negative-leaning, positive (as in fully discounting negative utility – I don’t think anyone actually holds this), ‘middling’ ie ‘normal’ (often called positive, but it would benefit from a distinct adjective)
• What’s our primary mentality toward it? (mentality axis); eg act, rule, two-level, global
• How do we deal with changing populations? (population axis); eg average, total
• To what extent do we discount future utility? (discounting axis); eg zero discount, >0 discount
• How do we pinpoint the net zero utility point? (balancing axis); eg Tännsjö’s test, experience tradeoffs
• What is a utilon? (utilon axis) [3] – I don’t know of any examples of serious discussion on this (other than generic dismissals of the question), but it’s ultimately a question utilitarians will need to answer if they wish to formalise their system.

For preference utilitarianism in particular:

Here then, are the six most salient dependent axes of preference utilitarianism, ie those that describe what could count as utility for PUs. I’ll refer to the poles on each axis as (axis)0 and (axis)1, where any intermediate view will be (axis)X. We can then formally refer to subtypes, and also exclude them, eg ~(F0)R1PU, or ~(F0 v R1)PU etc, or represent a range, eg C0..XPU.

How do we process misinformed preferences? (information axis F)

(F0 no adjustment / F1 adjust to what it would have been had the person been fully informed / FX somewhere in between)

How do we process irrational preferences? (rationality axis R)

(R0 no adjustment / R1 adjust to what it would have been had the person been fully rational / RX somewhere in between)

How do we process malformed preferences? (malformation axes M)

(M0 Ignore them / MF1 adjust to fully informed / MFR1 adjust to fully informed and rational (shorthand for MF1R1) / MFxRx adjust to somewhere in between)

How long is a preference relevant? (duration axis D)

(D0 During its expression only / DF1 During and future / DPF1 During, future and past (shorthand for  DP1F1) / DPxFx Somewhere in between)

What constitutes a preference? (constitution axis C)

(C0 Phenomenal experience only / C1 Behaviour only / CX A combination of the two)

What resolves a preference? (resolution axis S)

(S0 Phenomenal experience only / S1 External circumstances only / SX A combination of the two)

What distinguishes these categorisations is that each category, as far as I can perceive, has no analogous axis within hedonistic utilitarianism. In other words to a hedonistic utilitarian, such axes would either be meaningless, or have only one logical answer. But any well-defined and consistent form of preference utilitarianism must sit at some point on every one of these axes.

See the article for more detailed discussion about each of the axes of preference utilitarianism, and more.

I was recently re-reading a piece by Yvain/Scott Alexander called Epistemic Learned Helplessness. It's a very insightful post, as is typical for Scott, and I recommend giving it a read if you haven't already. In it he writes:

When I was young I used to read pseudohistory books; Immanuel Velikovsky's Ages in Chaos is a good example of the best this genre has to offer. I read it and it seemed so obviously correct, so perfect, that I could barely bring myself to bother to search out rebuttals.

And then I read the rebuttals, and they were so obviously correct, so devastating, that I couldn't believe I had ever been so dumb as to believe Velikovsky.

And then I read the rebuttals to the rebuttals, and they were so obviously correct that I felt silly for ever doubting.

And so on for several more iterations, until the labyrinth of doubt seemed inescapable.

He goes on to conclude that the skill of taking ideas seriously - often considered one of the most important traits a rationalist can have - is a dangerous one. After all, it's very easy for arguments to sound convincing even when they're not, and if you're too easily swayed by argument you can end up with some very absurd beliefs (like that Venus is a comet, say).

This post really resonated with me. I've had several experiences similar to what Scott describes, of being trapped between two debaters who both had a convincingness that exceeded my ability to discern truth. And my reaction in those situations was similar to his: eventually, after going through the endless chain of rebuttals and counter-rebuttals, changing my mind at each turn, I was forced to throw up my hands and admit that I probably wasn't going to be able to determine the truth of the matter - at least, not without spending a lot more time investigating the different claims than I was willing to. And so in many cases I ended up adopting a sort of semi-principled stance of agnosticism: unless it was a really really important question (in which case I was sort of obligated to do the hard work of investigating the matter to actually figure out the truth), I would just say I don't know when asked for my opinion.

[Non-exhaustive list of areas in which I am currently epistemically helpless: geopolitics (in particular the Israel/Palestine situation), anthropics, nutrition science, population ethics]

All of which is to say: I think Scott is basically right here, in many cases we shouldn't have too strong of an opinion on complicated matters. But when I re-read the piece recently I was struck by the fact that his whole argument could be summed up much more succinctly (albeit much more pithily) as:

"Don't be gullible."

Huh. Sounds a lot more obvious that way.

Now, don't get me wrong: this is still good advice. I think people should endeavour to not be gullible if at all possible. But it makes you wonder: why did Scott feel the need to write a post denouncing gullibility? After all, most people kind of already think being gullible is bad - who exactly is he arguing against here?

Well, recall that he wrote the post in response to the notion that people should believe arguments and take ideas seriously. These sound like good, LW-approved ideas, but note that unless you're already exceptionally smart or exceptionally well-informed, believing arguments and taking ideas seriously is tantamount to...well, to being gullible. In fact, you could probably think of gullibility as a kind of extreme and pathological form of lightness; a willingness to be swept away by the winds of evidence, no matter how strong (or weak) they may be.

There seems to be some tension here. On the one hand we have an intuitive belief that gullibility is bad; that the proper response to any new claim should be skepticism. But on the other hand we also have some epistemic norms here at LW that are - well, maybe they don't endorse being gullible, but they don't exactly not endorse it either. I'd say the LW memeplex is at least mildly friendly towards the notion that one should believe conclusions that come from convincing-sounding arguments, even if they seem absurd. A core tenet of LW is that we change our mind too little, not too much, and we're certainly all in favour of lightness as a virtue.

Anyway, I thought about this tension for a while and came to the conclusion that I had probably just lost sight of my purpose. The goal of (epistemic) rationality isn't to not be gullible or not be skeptical - the goal is to form correct beliefs, full stop. Terms like gullibility and skepticism are useful to the extent that people tend to be systematically overly accepting or dismissive of new arguments - individual beliefs themselves are simply either right or wrong. So, for example, if we do studies and find out that people tend to accept new ideas too easily on average, then we can write posts explaining why we should all be less gullible, and give tips on how to accomplish this. And if on the other hand it turns out that people actually accept far too few new ideas on average, then we can start talking about how we're all much too skeptical and how we can combat that. But in the end, in terms of becoming less wrong, there's no sense in which gullibility would be intrinsically better or worse than skepticism - they're both just words we use to describe deviations from the ideal, which is accepting only true ideas and rejecting only false ones.

This answer basically wrapped the matter up to my satisfaction, and resolved the sense of tension I was feeling. But afterwards I was left with an additional interesting thought: might gullibility be, if not a desirable end point, then an easier starting point on the path to rationality?

That is: no one should aspire to be gullible, obviously. That would be aspiring towards imperfection. But if you were setting out on a journey to become more rational, and you were forced to choose between starting off too gullible or too skeptical, could gullibility be an easier initial condition?

I think it might be. It strikes me that if you start off too gullible you begin with an important skill: you already know how to change your mind. In fact, changing your mind is in some ways your default setting if you're gullible. And considering that like half the freakin sequences were devoted to learning how to actually change your mind, starting off with some practice in that department could be a very good thing.

I consider myself to be...well, maybe not more gullible than average in absolute terms - I don't get sucked into pyramid scams or send money to Nigerian princes or anything like that. But I'm probably more gullible than average for my intelligence level. There's an old discussion post I wrote a few years back that serves as a perfect demonstration of this (I won't link to it out of embarrassment, but I'm sure you could find it if you looked). And again, this isn't a good thing - to the extent that I'm overly gullible, I aspire to become less gullible (Tsuyoku Naritai!). I'm not trying to excuse any of my past behaviour. But when I look back on my still-ongoing journey towards rationality, I can see that my ability to abandon old ideas at the (relative) drop of a hat has been tremendously useful so far, and I do attribute that ability in part to years of practice at...well, at believing things that people told me, and sometimes gullibly believing things that people told me. Call it epistemic deferentiality, or something - the tacit belief that other people know better than you (especially if they're speaking confidently) and that you should listen to them. It's certainly not a character trait you're going to want to keep as a rationalist, and I'm still trying to do what I can to get rid of it - but as a starting point? You could do worse I think.

Now, I don't pretend that the above is anything more than a plausibility argument, and maybe not a strong one at that. For one I'm not sure how well this idea carves reality at its joints - after all, gullibility isn't quite the same thing as lightness, even if they're closely related. For another, if the above were true, you would probably expect LWer's to be more gullible than average. But that doesn't seem quite right - while LW is admirably willing to engage with new ideas, no matter how absurd they might seem, the default attitude towards a new idea on this site is still one of intense skepticism. Post something half-baked on LW and you will be torn to shreds. Which is great, of course, and I wouldn't have it any other way - but it doesn't really sound like the behaviour of a website full of gullible people.

(Of course, on the other hand it could be that LWer's really are more gullible than average, but they're just smart enough to compensate for it)

Anyway, I'm not sure what to make of this idea, but it seemed interesting and worth a discussion post at least. I'm curious to hear what people think: does any of the above ring true to you? How helpful do you think gullibility is, if it is at all? Can you be "light" without being gullible? And for the sake of collecting information: do you consider yourself to be more or less gullible than average for someone of your intelligence level?