Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Aumann Agreement Game

5 abramdemski 09 October 2015 05:14PM

I've written up a rationality game which we played several times at our local LW chapter and had a lot of fun with. The idea is to put Aumann's agreement theorem into practice as a multi-player calibration game, in which players react to the probabilities which other players give (each holding some privileged evidence). If you get very involved, this implies reasoning not only about how well your friends are calibrated, but also how much your friends trust each other's calibration, and how much they trust each other's trust in each other.

You'll need a set of trivia questions to play. We used these

The write-up includes a helpful scoring table which we have not play-tested yet. We did a plain Bayes loss rather than an adjusted Bayes loss when we played, and calculated things on our phone calculators. This version should feel a lot better, because the numbers are easier to interpret and you get your score right away rather than calculating at the end.

New LW Meetups: Reading, Stockholm, and Suzhou

1 FrankAdamek 09 October 2015 04:11PM

This summary was posted to LW Main on October 2nd. The following week's summary is here.

New meetups (or meetups with a hiatus of more than a year) are happening in:

Irregularly scheduled Less Wrong meetups are taking place in:

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Cambridge UK, Canberra, Columbus, Denver, London, Madison WI, Melbourne, Moscow, Mountain View, New Hampshire, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

continue reading »

Toy model for wire-heading

2 Stuart_Armstrong 09 October 2015 03:45PM

A putative new idea for AI control; index here.

This is a (very) simple toy model of the wire-heading problem to illustrate how it might or might not happen. The great question is "where do we add the (super)intelligence?"

Let's assume a simple model for an expected utility maximising agent. There's the input assessor module A, which takes various inputs and computes the agent's "reward" or "utility". For a reward-based agent, A is typically outside of the agent; for a utility-maximiser, it's typically inside the agent, though the distinction need not be sharp. And there's the the decision module D, which assess the possible actions to take to maximise the output of A. If E is the general environment, we have D+A+E.

Now let's make the agent superintelligent. If we add superintelligence to module D, then D will wirehead by taking control of A (whether A is inside the agent or not) and controlling E to prevent interference. If we add superintelligence to module A, then it will attempt to compute rewards as effectively as possible, sacrificing D and E to achieve it's efficient calculations.

Therefore to prevent wireheading, we need to "add superintelligence" to (D+A), making sure that we aren't doing so to some sub-section of the algorithm - which might be hard if the "superintelligence" is obscure or black-box.

Philosophical schools are approaches not positions

2 casebash 09 October 2015 09:46AM

One of the great challenges of learning philosophy is trying to understand the difference between different schools of thought. Often it can be almost impossible to craft a definition that is specific enough to be understandable, whilst also being general enough to convey to breadth of that school of thought. I would suggest that this is a result of trying to define a school as taking a particular position in a debate, when they would be better defined as taking a particular approach to answering a question.

Take for example dualism and monism. Dualists believe that there exist two substances (typically the material substance and some kind of soul/consciousness), while monists believe that there only exists one. The question of whether this debate is defined precisely enough to actually be answerable immediately crops up. Few people would object to labelling the traditional Christian model of souls which went to an afterlife as being a Dualist model or a model of our universe with no conscious beings whatsoever as being monist. However, providing a good, general definition of what would count as two substances and what would count as one seems extraordinarily difficult. The question then arises of whether the dualism vs. monism debate is actually in a form that is answerable.

In contrast, if Dualism and Monism are thought of as approaches, then there can conceivably exist some situations Dualism is clearly better, some situations where Monism is clearly better and some situations where it is debatable. Rather than labelling the situation to be unanswerable, it would be better to call it possibly unanswerable.

Once it is accepted that dualism and monism are approaches, rather than positions the debate becomes much clearer. We can define these approaches as follows: Monism argues for describing reality as containing a single substance, while dualism argues for describing reality as containing two substances: typically one being physical and the other being mental or spiritual. I originally wrote this sentence using the word ‘modelling’ instead of ‘describing’, but I changed it because I wanted to be neutral on the issue on whether we can talk about what actually exists or can only talk about models of reality. If it was meaningful to talk about whether one or two substances actually existed (as opposed to simply being useful models), then the monism and dualism approaches would collapse down to being positions. However, the assumption that they have a "real" existence, if that is actually a valid concept, should not be made at the outset, and hence we describe them as approaches.

Can we still have our dualism vs. monism debate? Sure, kind of. We begin by using philosophy to establish the facts. In some cases, only one description may match the situation, but in other cases, it may be ambiguous. If this occurs, we could allow a debate to occur over which is the better description . This seems like a positional debate, but simply understanding that it is a descriptional debate changes how the debate plays out. Some people would argue that this question isn’t a job for philosophers, but for linguists, and I acknowledge that's there's a lot of validity to this point of view. Secondly, these approaches could be crystalised into actual positions. This would involve creating criteria for one side to win and the other to lose. Many philosophers who belong to monism, for example, would dislike the "crystalised" monism for not representing their name, so it might be wise to give these crystilised positions their own name.

We also consider free will. Instead of understanding the free will school of philosophy to hold the position that F0 exists where F0 is what is really meant by free will, it is better to understand it as an general approach that argues that there is some aspect of reality accurately described by the phrase “free will”. Some people will find this definition unsatisfactory and almost tauntological, but no more precise statement can be made if we want to capture the actual breadth of thought. If you want to know what this person actually believes, then you’ll have to ask them to define what they are using free will to mean.

This discussion also leads us a better way to teach people about these terms. The first part is to explain how the particular approach tries to describe reality. The second is to explain why particular situations or thought experiments seems to make more sense with this description.

While I have maintained that philosophical schools should be understood as approaches, rather than positions, I admit the possibility than in a few cases philosophers might have actually managed to come to consensus and make the opposing schools of thought positions rather than approaches. This analysis would not apply to them. However, if these cases do in fact exist, the appear to be far and few between.

Note: I'm not completely happy with the monism, dualism example, I'll probably replace it later when I come across a better example for demonstrating my point.

Emotional tools for the beginner rationalist

1 Gleb_Tsipursky 09 October 2015 05:01AM

Something that I haven't seen really discussed is what kind of emotional tools would be good for beginner rationalists. I'm especially interested in this topic since as part of my broader project of spreading rationality to a wide audience and thus raising the sanity waterline, I come across a lot of people who are interested in becoming more rational, but have difficulty facing the challenges of the Valley of Bad Rationality. In other words, they have trouble acknowledging their own biases and faults, facing the illusions within their moral systems and values, letting go of cached patterns, updating their beliefs, etc. Many thus abandon their aspiration toward rationality before they get very far. I think this is a systematic failure mode of many beginner aspiring rationalists, and so I wanted to start a discussion about what we can do about it as a community.


Note that this emotional danger does not feel intuitive to me or likely to many of you. In a Facebook discussion with Viliam Bur, he pointed out how he did not experience the Valley. I personally did not experience it that much either. However, based on the evidence of the Intentional Insights outreach efforts, this is a typical mind fallacy particular to many but far from all aspiring rationalists. So we should make an effort to address it in order to raise the sanity waterline effectively.


I'll start by sharing what I found effective in my own outreach efforts. First, I found it helpful to frame the aspiration toward rationality not as a search for a perfect and unreachable ideal, but as a way of constant improvement from the baseline where all humans are to something better. I highlight the benefits people get from this improved mode of thinking, to prime people to focus on their current self and detach themselves from their past selves. I highlight the value of self-empathy and self-forgiveness toward oneself for holding mistaken views, and encourage people to think of themselves as becoming more right, rather than less wrong :-)


Another thing that I found helpful was to provide new aspiring rationalists with a sense of community and social belonging. Joining a community of aspiring rationalists who are sensitive toward a newcomers' emotions, and help that newcomer deal with the challenges s/he experiences, is invaluable for overcoming the emotional strains of the Valley. Something especially useful is having people who are trained in coaching/counseling and serve as mentors for new members, who can help be guides for their intellectual and emotional development alike. I'd suggest that every LW meetup group consider instituting a system of mentors who can provide emotional and intellectual support alike for new members.


Now I'd like to hear about your experiences traveling the Valley, and what tools you and others you know used to manage it. Also, what are your ideas about useful tools for that purpose in general? Look forward to hearing your thoughts!




[Link] Stephen Hawking AMA answers

5 AspiringRationalist 08 October 2015 11:13PM

Fiction Considered Harmful

5 abramdemski 08 October 2015 06:34PM

Epistemic status: playing devil's advocate.

I wrote the following a couple of weeks back for a meet-up post, and Gunnar_Zarncke suggested I should turn it into a discussion post:

continue reading »

Rationality Reading Group: Part K: Letting Go

3 Gram_Stone 08 October 2015 02:32AM

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Part K: Letting Go (pp. 497-532)This post summarizes each article of the sequence, linking to the original LessWrong post where available.

K. Letting Go

121. The Importance of Saying "Oops" - When your theory is proved wrong, just scream "OOPS!" and admit your mistake fully. Don't just admit local errors. Don't try to protect your pride by conceding the absolute minimal patch of ground. Making small concessions means that you will make only small improvements. It is far better to make big improvements quickly. This is a lesson of Bayescraft that Traditional Rationality fails to teach.

122. The Crackpot Offer - If you make a mistake, don't excuse it or pat yourself on the back for thinking originally; acknowledge you made a mistake and move on. If you become invested in your own mistakes, you'll stay stuck on bad ideas.

123. Just Lose Hope Already - Casey Serin owes banks 2.2 million dollars after lying on mortgage applications in order to simultaneously buy 8 different houses in different states. The sad part is that he hasn't given up - he hasn't declared bankruptcy, and has just attempted to purchase another house. While this behavior seems merely stupid, it brings to mind Merton and Scholes of Long-Term Capital Management, who made 40% profits for three years, and then lost it all when they overleveraged. Each profession has rules on how to be successful, which makes rationality seem unlikely to help greatly in life. Yet it seems that one of the greater skills is not being stupid, which rationality does help with.

124. The Proper Use of Doubt - Doubt is often regarded as virtuous for the wrong reason: because it is a sign of humility and recognition of your place in the hierarchy. But from a rationalist perspective, this is not why you should doubt. The doubt, rather, should exist to annihilate itself: to confirm the reason for doubting, or to show the doubt to be baseless. When you can no longer make progress in this respect, the doubt is no longer useful to you as a rationalist.

125. You Can Face Reality - This post quotes a poem by Eugene Gendlin, which reads, "What is true is already so. / Owning up to it doesn't make it worse. / Not being open about it doesn't make it go away. / And because it's true, it is what is there to be interacted with. / Anything untrue isn't there to be lived. / People can stand what is true, / for they are already enduring it."

126. The Meditation on Curiosity - If you can find within yourself the slightest shred of true uncertainty, then guard it like a forester nursing a campfire. If you can make it blaze up into a flame of curiosity, it will make you light and eager, and give purpose to your questioning and direction to your skills.

127. No One Can Exempt You From Rationality's Laws - Traditional Rationality is phrased in terms of social rules, with violations interpretable as cheating - as defections from cooperative norms. But viewing rationality as a social obligation gives rise to some strange ideas. The laws of rationality are mathematics, and no social maneuvering can exempt you.

128. Leave a Line of Retreat - If you are trying to judge whether some unpleasant idea is true you should visualise what the world would look like if it were true, and what you would do in that situation. This will allow you to be less scared of the idea, and reason about it without immediately trying to reject it.

129. Crisis of Faith - A guide to making drastic but desirable changes to your map of the territory without losing your way.

130. The Ritual - Depiction of crisis of faith in Beisutsukai world.


This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

The next reading will cover Minds: An Introduction (pp. 539-545), Interlude: The Power of Intelligence (pp. 547-550), and Part L: The Simple Math of Evolution (pp. 553-613). The discussion will go live on Wednesday, 21 October 2015, right here on the discussion forum of LessWrong.

Group Rationality Diary, Oct. 6-18, 2015

4 polymathwannabe 06 October 2015 11:23PM

This is the public group rationality diary for October 6-18, 2015. It's a place to record and chat about it if you have done, or are actively doing, things like:

  • Established a useful new habit

  • Obtained new evidence that made you change your mind about some belief

  • Decided to behave in a different way in some set of situations

  • Optimized some part of a common routine or cached behavior

  • Consciously changed your emotions or affect with respect to something

  • Consciously pursued new valuable information about something that could make a big difference in your life

  • Learned something new about your beliefs, behavior, or life that surprised you

  • Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Crazy Ideas Thread - October 2015

5 Gunnar_Zarncke 06 October 2015 10:38PM

This thread is intended to provide a space for 'crazy' ideas. Ideas that spontaneously come to mind (and feel great), ideas you long wanted to tell but never found the place and time for and also for ideas you think should be obvious and simple - but nobody ever mentions them. 

Rules for this thread:

  1. Each crazy idea goes into its own top level comment and may be commented there.
  2. Voting should be based primarily on how original the idea is.
  3. Meta discussion of the thread should go to the top level comment intended for that purpose. 


If you create such a thread do the following :

  • Use "Crazy Ideas Thread" in the title.
  • Copy the rules.
  • Add the tag "crazy_idea".
  • Create a top-level comment saying 'Discussion of this thread goes here; all other top-level comments should be ideas or similar'
  • Add a second top-level comment with an initial crazy idea to start participation.

A few misconceptions surrounding Roko's basilisk

37 RobbBB 05 October 2015 09:23PM

There's a new LWW page on the Roko's basilisk thought experiment, discussing both Roko's original post and the fallout that came out of Eliezer Yudkowsky banning the topic on Less Wrong discussion threads. The wiki page, I hope, will reduce how much people have to rely on speculation or reconstruction to make sense of the arguments.

While I'm on this topic, I want to highlight points that I see omitted or misunderstood in some online discussions of Roko's basilisk. The first point that people writing about Roko's post often neglect is:


  • Roko's arguments were originally posted to Less Wrong, but they weren't generally accepted by other Less Wrong users.

Less Wrong is a community blog, and anyone who has a few karma points can post their own content here. Having your post show up on Less Wrong doesn't require that anyone else endorse it. Roko's basic points were promptly rejected by other commenters on Less Wrong, and as ideas not much seems to have come of them. People who bring up the basilisk on other sites don't seem to be super interested in the specific claims Roko made either; discussions tend to gravitate toward various older ideas that Roko cited (e.g., timeless decision theory (TDT) and coherent extrapolated volition (CEV)) or toward Eliezer's controversial moderation action.

In July 2014, David Auerbach wrote a Slate piece criticizing Less Wrong users and describing them as "freaked out by Roko's Basilisk." Auerbach wrote, "Believing in Roko’s Basilisk may simply be a 'referendum on autism'" — which I take to mean he thinks a significant number of Less Wrong users accept Roko’s reasoning, and they do so because they’re autistic (!). But the Auerbach piece glosses over the question of how many Less Wrong users (if any) in fact believe in Roko’s basilisk. Which seems somewhat relevant to his argument...?

The idea that Roko's thought experiment holds sway over some community or subculture seems to be part of a mythology that’s grown out of attempts to reconstruct the original chain of events; and a big part of the blame for that mythology's existence lies on Less Wrong's moderation policies. Because the discussion topic was banned for several years, Less Wrong users themselves had little opportunity to explain their views or address misconceptions. A stew of rumors and partly-understood forum logs then congealed into the attempts by people on RationalWiki, Slate, etc. to make sense of what had happened.

I gather that the main reason people thought Less Wrong users were "freaked out" about Roko's argument was that Eliezer deleted Roko's post and banned further discussion of the topic. Eliezer has since sketched out his thought process on Reddit:

When Roko posted about the Basilisk, I very foolishly yelled at him, called him an idiot, and then deleted the post. [...] Why I yelled at Roko: Because I was caught flatfooted in surprise, because I was indignant to the point of genuine emotional shock, at the concept that somebody who thought they'd invented a brilliant idea that would cause future AIs to torture people who had the thought, had promptly posted it to the public Internet. In the course of yelling at Roko to explain why this was a bad thing, I made the further error---keeping in mind that I had absolutely no idea that any of this would ever blow up the way it did, if I had I would obviously have kept my fingers quiescent---of not making it absolutely clear using lengthy disclaimers that my yelling did not mean that I believed Roko was right about CEV-based agents [= Eliezer’s early model of indirectly normative agents that reason with ideal aggregated preferences] torturing people who had heard about Roko's idea. [...] What I considered to be obvious common sense was that you did not spread potential information hazards because it would be a crappy thing to do to someone. The problem wasn't Roko's post itself, about CEV, being correct.

This, obviously, was a bad strategy on Eliezer's part. Looking at the options in hindsight: To the extent it seemed plausible that Roko's argument could be modified and repaired, Eliezer shouldn't have used Roko's post as a teaching moment and loudly chastised him on a public discussion thread. To the extent this didn't seem plausible (or ceased to seem plausible after a bit more analysis), continuing to ban the topic was a (demonstrably) ineffective way to communicate the general importance of handling real information hazards with care.


On that note, point number two:

  • Roko's argument wasn’t an attempt to get people to donate to Friendly AI (FAI) research. In fact, the opposite is true.

Roko's original argument was not 'the AI agent will torture you if you don't donate, therefore you should help build such an agent'; his argument was 'the AI agent will torture you if you don't donate, therefore we should avoid ever building such an agent.' As Gerard noted in the ensuing discussion thread, threats of torture "would motivate people to form a bloodthirsty pitchfork-wielding mob storming the gates of SIAI [= MIRI] rather than contribute more money." To which Roko replied: "Right, and I am on the side of the mob with pitchforks. I think it would be a good idea to change the current proposed FAI content from CEV to something that can't use negative incentives on x-risk reducers."

Roko saw his own argument as a strike against building the kind of software agent Eliezer had in mind. Other Less Wrong users, meanwhile, rejected Roko's argument both as a reason to oppose AI safety efforts and as a reason to support AI safety efforts.

Roko's argument was fairly dense, and it continued into the discussion thread. I’m guessing that this (in combination with the temptation to round off weird ideas to the nearest religious trope, plus misunderstanding #1 above) is why RationalWiki's version of Roko’s basilisk gets introduced as

a futurist version of Pascal’s wager; an argument used to try and suggest people should subscribe to particular singularitarian ideas, or even donate money to them, by weighing up the prospect of punishment versus reward.

If I'm correctly reconstructing the sequence of events: Sites like RationalWiki report in the passive voice that the basilisk is "an argument used" for this purpose, yet no examples ever get cited of someone actually using Roko’s argument in this way. Via citogenesis, the claim then gets incorporated into other sites' reporting.

(E.g., in Outer Places: "Roko is claiming that we should all be working to appease an omnipotent AI, even though we have no idea if it will ever exist, simply because the consequences of defying it would be so great." Or in Business Insider: "So, the moral of this story: You better help the robots make the world a better place, because if the robots find out you didn’t help make the world a better place, then they’re going to kill you for preventing them from making the world a better place.")

In terms of argument structure, the confusion is equating the conditional statement 'P implies Q' with the argument 'P; therefore Q.' Someone asserting the conditional isn’t necessarily arguing for Q; they may be arguing against P (based on the premise that Q is false), or they may be agnostic between those two possibilities. And misreporting about which argument was made (or who made it) is kind of a big deal in this case: 'Bob used a bad philosophy argument to try to extort money from people' is a much more serious charge than 'Bob owns a blog where someone once posted a bad philosophy argument.'



  • "Formally speaking, what is correct decision-making?" is an important open question in philosophy and computer science, and formalizing precommitment is an important part of that question.

Moving past Roko's argument itself, a number of discussions of this topic risk misrepresenting the debate's genre. Articles on Slate and RationalWiki strike an informal tone, and that tone can be useful for getting people thinking about interesting science/philosophy debates. On the other hand, if you're going to dismiss a question as unimportant or weird, it's important not to give the impression that working decision theorists are similarly dismissive.

What if your devastating take-down of string theory is intended for consumption by people who have never heard of 'string theory' before? Even if you're sure string theory is hogwash, then, you should be wary of giving the impression that the only people discussing string theory are the commenters on a recreational physics forum. Good reporting by non-professionals, whether or not they take an editorial stance on the topic, should make it obvious that there's academic disagreement about which approach to Newcomblike problems is the right one. The same holds for disagreement about topics like long-term AI risk or machine ethics.

If Roko's original post is of any pedagogical use, it's as an unsuccessful but imaginative stab at drawing out the diverging consequences of our current theories of rationality and goal-directed behavior. Good resources for these issues (both for discussion on Less Wrong and elsewhere) include:

The Roko's basilisk ban isn't in effect anymore, so you're welcome to direct people here (or to the Roko's basilisk wiki page, which also briefly introduces the relevant issues in decision theory) if they ask about it. Particularly low-quality discussions can still get deleted (or politely discouraged), though, at moderators' discretion. If anything here was unclear, you can ask more questions in the comments below.

How could one (and should one) convert someone from pseudoscience?

7 Vilx- 05 October 2015 11:53AM

I've known for a long time that some people who are very close to me are somewhat inclined to believe the pseudoscience world, but it always seemed pretty benign. In their everyday lives they're pretty normal people and don't do any crazy things, so this was a topic I mostly avoided and left it at that. After all - they seemed to find psychological value in it. A sense of control over their own lives, a sense of purpose, etc.

Recently I found out however that at least one of them seriously believes Bruce Lipton, who in essence preaches that happy thoughts cure cancer. Now I'm starting to get worried...

Thus I'm wondering - what can I do about it? This is in essence a religious question. They believe this stuff with just anecdotal proof. How do I disprove it without sounding like "Your religion is wrong, convert to my religion, it's right"? Pseudoscientists are pretty good at weaving a web of lies that sound quite logical and true.

The one thing I've come up with is to somehow introduce them to classical logical fallacies. That at least doesn't directly conflict with their beliefs. But beyond that I have no idea.

And perhaps more important is the question - should I do anything about it? The pseudoscientific world is a rosy one. You're in control of your life and your body, you control random events, and most importantly - if you do everything right, it'll all be OK. Even if I succeed in crushing that illusion, I have nothing to put in its place. I'm worried that revealing just how truly bleak the reality is might devastate them. They seem to be drawing a lot of their happiness from these pseudoscientific beliefs, either directly or indirectly.

And anyway, more likely that I won't succeed but just ruin my (healthy) relationship with them. Maybe it's best just not to interfere at all? Even if they end up hurting themselves, well... it was their choice. Of course, that also means that I'll be standing idly by and allowing bullshit to propagate, which is kinda not a very good thing. However right now they are not very pushy about their beliefs, and only talk about them if the topic comes up naturally, so I guess it's not that bad.

Any thoughts?

Open thread, Oct. 5 - Oct. 11, 2015

7 MrMind 05 October 2015 06:50AM

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

[Link] Tetlock on the power of precise predictions to counter political polarization

5 Stefan_Schubert 04 October 2015 03:19PM

The prediction expert Philip Tetlock writes in New York Times on the power of precise predictions to counter political polarization. Note the similarity to Robin Hanson's futarchy idea.

IS there a solution to this country’s polarized politics?

Consider the debate over the nuclear deal with Iran, which was one of the nastiest foreign policy fights in recent memory. There was apocalyptic rhetoric, multimillion-dollar lobbying on both sides and a near-party-line Senate vote. But in another respect, the dispute was hardly unique: Like all policy debates, it was, at its core, a contest between competing predictions.

Opponents of the deal predicted that the agreement would not prevent Iran from getting the bomb, would put Israel at greater risk and would further destabilize the region. The deal’s supporters forecast that it would stop (or at least delay) Iran from fielding a nuclear weapon, would increase security for the United States and Israel and would underscore American leadership.

The problem with such predictions is that it is difficult to square them with objective reality. Why? Because few of them are specific enough to be testable. Key terms are left vague and undefined. (What exactly does “underscore leadership” mean?) Hedge words like “might” or “could” are deployed freely. And forecasts frequently fail to include precise dates or time frames. Even the most emphatic declarations — like former Vice President Dick Cheney’s prediction that the deal “will lead to a nuclear-armed Iran” — can be too open-ended to disconfirm.


Non-falsifiable predictions thus undermine the quality of our discourse. They also impede our ability to improve policy, for if we can never judge whether a prediction is good or bad, we can never discern which ways of thinking about a problem are best.

The solution is straightforward: Replace vague forecasts with testable predictions. Will the International Atomic Energy Agency report in December that Iran has adequately resolved concerns about the potential military dimensions of its nuclear program? Will Iran export or dilute its quantities of low-enriched uranium in excess of 300 kilograms by the deal’s “implementation day” early next year? Within the next six months, will any disputes over I.A.E.A. access to Iranian sites be referred to the Joint Commission for resolution?

Such questions don’t precisely get at what we want to know — namely, will the deal make the United States and its allies safer? — but they are testable and relevant to the question of the Iranian threat. Most important, they introduce accountability into forecasting. And that, it turns out, can depolarize debate.

In recent years, Professor Tetlock and collaborators have observed this depolarizing effect when conducting forecasting “tournaments” designed to identify what separates good forecasters from the rest of us. In these tournaments, run at the behest of the Intelligence Advanced Research Projects Activity (which supports research relevant to intelligence agencies), thousands of forecasters competed to answer roughly 500 questions on various national security topics, from the movement of Syrian refugees to the stability of the eurozone.

The tournaments identified a small group of people, the top 2 percent, who generated forecasts that, when averaged, beat the average of the crowd by well over 50 percent in each of the tournament’s four years. How did they do it? Like the rest of us, these “superforecasters” have political views, often strong ones. But they learned to seriously consider the possibility that they might be wrong.

What made such learning possible was the presence of accountability in the tournament: Forecasters were able see their competitors’ predictions, and that transparency reduced overconfidence and the instinct to make bold, ideologically driven predictions. If you can’t hide behind weasel words like “could” or “might,” you start constructing your predictions carefully. This makes sense: Modest forecasts are more likely to be correct than bold ones — and no one wants to look stupid.

This suggests a way to improve real-world discussion. Suppose, during the next ideologically charged policy debate, that we held a public forecasting tournament in which representatives from both sides had to make concrete predictions. (We are currently sponsoring such a tournament on the Iran deal.) Based on what we have seen in previous tournaments, this exercise would decrease the distance between the two camps. And because it would be possible to determine a “winner,” it would help us learn whether the conservative or liberal assessment of the issue was more accurate.


Either way, we would begin to emerge from our dark age of political polarization.

Experiment: Changing minds vs. preaching to the choir

11 cleonid 03 October 2015 11:27AM


      1. Problem

In the market economy production is driven by monetary incentives – higher reward for an economic activity makes more people willing to engage in it. Internet forums follow the same principle but with a different currency - instead of money the main incentive of internet commenters is the reaction of their audience. A strong reaction expressed by a large number of replies or “likes” encourages commenters to increase their output. Its absence motivates them to quit posting or change their writing style.

On neutral topics, using audience reaction as an incentive works reasonably well: attention focuses on the most interesting or entertaining comments. However, on partisan issues, such incentives become counterproductive. Political forums and newspaper comment sections demonstrate the same patterns:

  • The easiest way to maximize “likes” for a given amount of effort is by posting an emotionally charged comment which appeals to audience’s biases (“preaching to the choir”).


  • The easiest way to maximize the number of replies is by posting a low quality comment that goes against audience’s biases (“trolling”).


  • Both effects are amplified when the website places comments with most replies or “likes” at the top of the page.


The problem is not restricted to low-brow political forums. The following graph, which shows the average number of comments as a function of an article’s karma, was generated from the Lesswrong data.


The data suggests that the easiest way to maximize the number of replies is to write posts that are disliked by most readers. For instance, articles with the karma of -1 on average generate twice as many comments (20.1±3.4) as articles with the karma of +1 (9.3±0.8).

2. Technical Solution

Enabling constructive discussion between people with different ideologies requires reversing the incentives – people need to be motivated to write posts that sound persuasive to the opposite side rather than to their own supporters.

We suggest addressing this problem that this problem by changing the voting system. In brief, instead of votes from all readers, comment ratings and position on the page should be based on votes from the opposite side only. For example, in the debate on minimum wage, for arguments against minimum wage only the upvotes of minimum wage supporters would be counted and vice versa.

The new voting system can simultaneously achieve several objectives:

·         eliminate incentives for preaching to the choir

·         give posters a more objective feedback on the impact of their contributions, helping them improve their writing style

·     focus readers’ attention on comments most likely to change their minds instead of inciting comments that provoke an irrational defensive reaction.

3. Testing

If you are interested in measuring and improving your persuasive skills and would like to help others to do the same, you are invited to take part in the following experiment:


Step I. Submit Pro or Con arguments on any of the following topics (up to 3 arguments in total):

     Should the government give all parents vouchers for private school tuition?

     Should developed countries increase the number of immigrants they receive?

     Should there be a government mandated minimum wage?


Step II. For each argument you have submitted, rate 15 arguments submitted by others.


Step III.  Participants will be emailed the results of the experiment including:

-         ratings their arguments receive from different reviewer groups (supporters, opponents and neutrals)

-         the list of the most persuasive Pro & Con arguments on each topic (i.e. arguments that received the highest ratings from opposing and neutral groups)

-         rating distribution in each group


Step IV (optional). If interested, sign up for the next round.


The experiment will help us test the effectiveness of the new voting system and develop the best format for its application.





Digital Immortality Map: How to collect enough information about yourself for future resurrection by AI

6 turchin 02 October 2015 10:21PM

If someone has died it doesn’t mean that you should stop trying to return him to life. There is one clear thing that you should do (after cryonics): collect as much information about the person as possible, as well as store his DNA sample, and hope that future AI will return him to life based on this information.


Two meanings of “Digital immortality”

The term “Digital immortality” is often confused with the notion of mind uploading, as the end result is almost the same: a simulated brain in a computer. https://en.wikipedia.org/wiki/Digital_immortality

But here, by the term “Digital immortality” I mean reconstruction of the person based on his digital footprint and other traces by future AI after this person death.

Mind uploading in the future will happen while the original is still alive (or while the brain exists in a frozen state) and will be connected to a computer by some kind of sophisticated interface, or the brain will be scanned. It cannot be done currently. 

On the other hand, reconstruction based on traces will be done by future AI. So we just need to leave enough traces and we could do it now.

But we don’t know how much traces are enough, so basically we should try to produce and preserve as many traces as possible. However, not all traces are equal in their predictive value. Some are almost random, and others are so common that they do not provide any new information about the person.


Cheapest way to immortality

Creating traces is an affordable way of reaching immortality. It could even be done for another person after his death, if we start to collect all possible information about him. 

Basically I am surprised that people don’t do it all the time. It could be done in a simple form almost for free and in the background – just start a video recording app on your notebook, and record everything into shared folder connected with a free cloud. (Evocam program for Mac is excellent, and mail.ru provides up 100gb free).

But really good digital immortality require 2-3 month commitment for self-description with regular every year updates. It may also require maximum several thousand dollars investment in durable disks, DNA testing, videorecorders, and free time to do it.

I understand how to set up this process and could help anyone interested.



The idea of personal identity is outside the scope of this map. I have another map on this topic (now in draft), I assume that the problem of personal identity will be solved in the future. Perhaps we will prove that information only is enough to solve the problem, or we will find that continuity of consciousness, but we will be able to construct mechanisms to transfer this identity independently of information. 

Digital immortality requires a very weak notion of identity. i.e. a model of behavior and thought processes is enough for an identity. This model may have some differences from the original, which I call “one night difference”, that is the typical difference between me-yesterday and me-today after one night's sleep. The meaningful part of this information has size from several megabytes to gigabits, but we may need to collect much more information as we can’t now extract meaningful part from random.

DI may also be based on even weaker notion of identity, that anyone who thinks that he is me, is me. Weaker notions of identity require less information to be preserved, and in last case it may be around 10K bytes (including name, indexical information and basic traits description)

But the question about the number of traces needed to create an almost exact model of a personality is still open. It also depends on predictive power of future AI: the stronger is AI, the less traces are enough.

Digital immortality is plan C in my Immortality Roadmap, where Plan A is life extension and Plan B is cryonics; it is not plan A, because it requires solving the identity problem plus the existence of powerful future AI.



I created my first version of it in the year 1990 when I was 16, immediately after I had finished school. It included association tables, drawings and lists of all people known to me, as well as some art, memoires, audiorecordings and encyclopedia od everyday objects around me.

There are several approaches to achieving digital immortality. The most popular one is passive that is simply videorecording of everything you do.

My idea was that a person can actively describe himself from inside. He may find and declare the most important facts about himself. He may run specific tests that will reveal hidden levels of his mind and sub consciousness. He can write a diary and memoirs. That is why I called my digital immortality project “self-description”.


Structure of the map

This map consists of two parts: theoretical and practical. The theoretical part lists basic assumptions and several possible approaches to reconstructing an individual, in which he is considered as a black box. If real neuron actions will become observable, the "box" will become transparent and real uploading will be possible.

There are several steps in the practical part:

- The first step includes all the methods of fixing information while the person of interest is alive.

- The second step is about preservation of the information.

- The third step is about what should be done to improve and promote the process.

- The final fourth step is about the reconstruction of the individual, which will be performed by AI after his death. In fact it may happen soon, may be in next 20-50 years.

There are several unknowns in DI, including the identity problem, the size and type of information required to create an exact model of the person, and the required power of future AI to operate the process. These and other problems are listed in the box on the right corner of the map.

The pdf of the map is here, and jpg is below.


Previous posts with maps:

Doomsday Argument Map

AGI Safety Solutions Map

A map: AI failures modes and levels

A Roadmap: How to Survive the End of the Universe

A map: Typology of human extinction risks

Roadmap: Plan of Action to Prevent Human Extinction Risks

Immortality Roadmap













Weekly LW Meetups

4 FrankAdamek 02 October 2015 04:22PM

This summary was posted to LW Main on September 25th. The following week's summary is here.

New meetups (or meetups with a hiatus of more than a year) are happening in:

Irregularly scheduled Less Wrong meetups are taking place in:

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Cambridge UK, Canberra, Columbus, Denver, London, Madison WI, Melbourne, Moscow, Mountain View, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

continue reading »

Donna Capsella and the four applicants, pt.1

0 Romashka 02 October 2015 02:15PM

Once upon a time, in a dark, cruel world – maybe a world darker and crueller than it is – there lived a woman who wanted a piece of the action. Her name was Capsella Medik, but we remember her as Donna Capsella. This is an anecdote from her youth, told by a man who lived to tell it.

...you've got to understand, Donna started small. Real small. No money, no allies, no kin, and her wiles were – as feminine as they are. Still, she was ambitious, even then, and she had to look the part.

Girl had a way with people. Here's how it went.

One night, she rents a room – one table, five chairs – and two armed bodies, and sets up a date with four men at once – Mr. Burr, Mr. Sapp, Mr. Ast and Mr. Oriss, who've never seen her before. All are single, thirty-ish white collars. One look at the guns, and they're no trouble at all.

On the table, there's a heap: a coloured picture, a box of beads, another box (empty), four stacks of paper, four pens, a calculator and a sealed envelope.

'So,' says Donna. 'I need a manager. A clever man who'd keep my bank happy while I am...abroad. I offer you to play a game – just one game – and the winner is going to sign these papers. You leave hired, or not at all.'

The game was based on Mendel's Laws – can you imagine? The police never stood a chance against her... She had it printed out – a kind of cheat-sheet. It's like, if you have some biological feature, it's either what your genes say, or you helped Nature along the way; and the exact – wording – can be different, so you have blue eyes or brown eyes. The wording is what they call allele. Some alleles, dominant, shout louder than others, recessive, so you'll have at most two copies of each gene (hopefully), but only one will ever be heard on the outside.

(It's not quite that simple, but we didn't protest. Guns, you know.)

So there was a picture of a plant whose leaves came in four shapes (made by two genes with two alleles each):


From left to right: simplex, rhomboidea, heteris and tenuis. Simplex had only recessive alleles, aabb. Rhomboidea and tenuis each had only one pair of recessive alleles – aaB? and A?bb. But heteris, that one was a puzzler: A?B?.

'Okay,' Donna waves her hand over the heap on the table. 'Here are the rules. You will see two parent plants, and then you will see their offspring – one at a time.' She shows us the box with the beads. 'Forty-eight kids total.' She begins putting some of the beads into the empty box, but we don't see which ones. 'The colours are like in the picture. You have to guess as much about the parents and the kids as you can as I go along. All betting stops when the last kid pops out. Guess wrong, even partially wrong, you lose a point, guess right, earn one. Screw around, you're out of the game. The one with the most points wins.'

'Uh,' mumbles Oriss. 'Can we, maybe, say we're not totally sure – ?..'

She smiles, and oh, those teeth. 'Yeah. Use your Bayes.'

And just like that, Oriss reaches to his stack of paper, ready to slog through all the calculations. (Oriss likes to go ahead and gamble based on some math, even if it's not rock solid yet.)

'Er,' tries Sapp. 'Do we have to share our guesses?'

'No, the others will only know that you earned or lost a point.'

And Sapp picks up his pen, but with a little frown. (He doesn't share much, does Sapp.)

'Um,' Ast breaks in. 'In a single round, do we guess simultaneously, or in some order?'

'Simultaneously. You write it down and give it to me.'

And Ast slumps down in his seat, sweating, and eyes the calculator. (Ast prefers to go where others lead, though he can change his mind lightning-fast.)

'Well,' Burr shrugs. 'I'll just follow rough heuristics, and we'll see how it goes.'

'Such as?' asks Donna, cocking her head to the side.

'As soon as there's a simplex kid, it all comes down to pure arithmetic, since we'll know both parents have at least one recessive allele for each of the genes. If both parents are heteris – and they will be, I see it in your eyes! – then the probability of at least one of them having at least one recessive allele is higher than the probability of neither having any. I can delay making guesses for a time and just learn what score the others get for theirs, since they're pretty easy to reverse-engineer – '

'What!' say Ast, Sapp and Oriss together.

'You won't get points fast enough,' Donna points out. 'You will lose.'

'I might lose. And you will hire me anyway. You need a clever man to keep your bank happy.'

Donna purses her lips.

'You haven't told anything of value, anything the others didn't know.'

'But of course,' Burr says humbly, and even the armed bodies scowl.

'You're only clever when you have someone to mooch off. I won't hire you alone.'


'Mind, I won't pick you if you lose too badly.'

Burr leers at her, and she swears under her breath.

'Enough,' says Donna and puts down two red beads – the parents – on the table.

We take our pens. She reaches out into the box of offspring.

The first bead is red.

And the second one is red.

And the third one is red.

...I tell you, it was the longest evening in my life.


So, what are your Fermi estimates for the numbers of points Mr. Burr, Mr. Sapp, Mr. Ast and Mr. Oriss each earned? And who was selected as a manager, or co-managers? And how many people left the room?

(I apologise - the follow-up won't be for a while.)

Two Growth Curves

30 AnnaSalamon 02 October 2015 12:59AM

Sometimes, it helps to take a model that part of you already believes, and to make a visual image of your model so that more of you can see it.

One of my all-time favorite examples of this: 

I used to often hesitate to ask dumb questions, to publicly try skills I was likely to be bad at, or to visibly/loudly put forward my best guesses in areas where others knew more than me.

I was also frustrated with this hesitation, because I could feel it hampering my skill growth.  So I would try to convince myself not to care about what people thought of me.  But that didn't work very well, partly because what folks think of me is in fact somewhat useful/important.

Then, I got out a piece of paper and drew how I expected the growth curves to go.

In blue, I drew the apparent-coolness level that I could achieve if I stuck with the "try to look good" strategy.  In brown, I drew the apparent-coolness level I'd have if I instead made mistakes as quickly and loudly as possible -- I'd look worse at first, but then I'd learn faster, eventually overtaking the blue line.

Suddenly, instead of pitting my desire to become smart against my desire to look good, I could pit my desire to look good now against my desire to look good in the future :)

I return to this image of two growth curves often when I'm faced with an apparent tradeoff between substance and short-term appearances.  (E.g., I used to often find myself scurrying to get work done, or to look productive / not-horribly-behind today, rather than trying to build the biggest chunks of capital for tomorrow.  I would picture these growth curves.)

October 2015 Media Thread

5 ArisKatsaris 01 October 2015 10:17PM

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.


  • Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.
  • If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.
  • Please post only under one of the already created subthreads, and never directly under the parent media thread.
  • Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.
  • Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.

Polling Thread - Tutorial

5 Gunnar_Zarncke 01 October 2015 09:47PM

After some hiatus another installment of the Polling Thread.

This is your chance to ask your multiple choice question you always wanted to throw in. Get qualified numeric feedback to your comments. Post fun polls.

Additionally this is your chance to learn to write polls. This installment is devoted to try out polls for the cautious and curious.

These are the rules:

  1. Each poll goes into its own top level comment and may be commented there.
  2. You must at least vote all polls that were posted earlier than you own. This ensures participation in all polls and also limits the total number of polls. You may of course vote without posting a poll.
  3. Your poll should include a 'don't know' option (to avoid conflict with 2). I don't know whether we need to add a troll catch option here but we will see.

If you don't know how to make a poll in a comment look at the Poll Markup Help.

This is a somewhat regular thread. If it is successful I may post again. Or you may. In that case do the following :

  • Use "Polling Thread" in the title.
  • Copy the rules.
  • Add the tag "poll".
  • Link to this Thread or a previous Thread.
  • Create a top-level comment saying 'Discussion of this thread goes here; all other top-level comments should be polls or similar'
  • Add a second top-level comment with an initial poll to start participation.

[Link] Differential Technology Development - Some Early Thinking

3 MattG 01 October 2015 02:08AM

This article gives a simple model to think about the positive effects of a friendly AI vs. the negative effects of an unfriendly AI, and let's you plug in certain assumptions to see if speeding up AI progress is worthwhile. Thought some of you here might be interested.


The Trolley Problem and Reversibility

7 casebash 30 September 2015 04:06AM

The most famous problem used when discussing consequentialism is that of the tram problem. A tram is hurtling towards the 5 people on the track, but if you flick a switch it will change tracks and kill only the one person instead. Utilitarians would say that you should flick the switch as it is better for there to be a single death than five. Some deontologists might agree with this, however, much more would object and argue that you don’t have the right to make that decision. This problem has different variations, such as one where you push someone in front of the train instead of them being on the track, but we’ll consider this one, as if it is accepted then it moves you a large way towards utilitarianism.

Let’s suppose that someone flicks the switch, but then realises the other side was actually correct and that they shouldn’t have flicked it. Do they now have an obligation to flick the switch back? What is interesting is that if they had just walked into the room and the train was heading towards the one person, they would have had an obligation *not* to flick the switch, but, having flicked it, it seems that they have an obligation to flick it back the other way.

Where this gets more puzzling is when we imagine Bob having observed Aaron flicking the switch? Arguably, if Aaron had no right to flick the switch, then Bob would have obligation to flick it back (or, if not an obligation, this would surely count as a moral good?). It is hard to argue against this conclusion, assuming that there is a strong moral obligation for Aaron not to flick the switch, along the lines of “Do not kill”. This logic seems consistent with how we act in other situations; if someone had tried to kill someone or steal something important from them; then most people would reverse or prevent the action if they could. 

But what if Aaron reveals that he was only flicking the switch because Cameron had flicked it first? Then Bob would be obligated to leave it alone, as Aaron would be doing what Bob was planning to do: prevent interference. We can also complicate it by imagining that a strong gust of wind was about to come and flick the switch, but Bob flicked it first. Is there now a duty to undo Bob's flick of the switch or does that fact that the switch was going to flick anyway abrogate that duty? This obligation to trace back the history seems very strange indeed. I can’t see any pathway to find a logical contradiction, but I can’t imagine that many people would defend this state of affairs.

But perhaps the key principle here is non-interference. When Aaron flicks the switch, he has interfered and so he arguably has the limited right to undo his interference. But when Bob decides to reverse this, perhaps this counts as interference also. So while Bob receives credit for preventing Aaron’s interference, this is outweighed by committing interference himself - acts are generally considered more important than omissions. This would lead to Bob being required to take no action, as there wouldn’t be any morally acceptable pathway with which to take action.

I’m not sure I find this line of thought convincing. If we don’t want anyone interfering with the situation, couldn’t we lock the switch in place before anyone (including Aaron) gets the chance or even the notion to interfere? It would seem rather strange to argue that we have to leave the door open to interference even before we know anyone is planning to do so. Next suppose that we don’t have glue, but we can install a mechanism that will flick the switch back if anyone tries to flick it. Principally, this doesn’t seem any different from installing glue.

Next, suppose we don’t have a machine to flick it back, so instead we install Bob. It seems that installing Bob is just as moral as installing an actual mechanism. It would seem rather strange to argue that “installing” Bob is moral, but any action he takes is immoral. There might be cases where “installing” someone is moral, but certain actions they take will be immoral. One example would be “installing” a policeman to enforce a law that is imperfect. We can expect the decision to hire the policeman to be moral if the law is general good, but, in certain circumstances, flaws in this law might make enforcement immoral. But here, we are imagining that *any* action Bob takes is immoral interference. It therefore seems strange to suggest that installing him could somehow be moral and so this line of thought seems to lead to a contradiction.

We consider one last situation: that we aren't allowed to interfere and that setting up a mechanism to stop interference also counts as interference. We first imagine that Obama has ordered a drone attack that is going to kill a (robot, just go with it) terrorist. He knows that the drone attack will cause collateral damage, but it will also prevent the terrorist from killing many more people on American soil. He wakes up the next morning and realises that he was wrong to violate the deontological principles, so he calls off the attack. Are there any deotologists who would argue that he doesn’t have the right to rescind his order? Rescinding the order does not seem to count as "further interference", instead it seems to count as "preventing his interference from occurring". Flicking the switch back seems functionally identical to rescinding the order. The train hasn’t hit the intersection; so there isn’t any casual entanglement, so it seems like flicking the switch is best characterised as preventing the interference from occurring. If we want to make the scenarios even more similar, we can imagine that flicking the switch doesn't force the train to go down one track or another, but instead orders the driver to take one particular track. It doesn't seem like changing this aspect of the problem should alter the morality at all.

This post has shown that deontological objections to the Trolley Problem tend to lead to non-obvious philosophical commitments that are not very well known. I didn't write this post so much as to try to show that deontology is wrong, as to start as conversation and help deontologists understand and refine their commitments better.

I also wanted to include one paragraph I wrote in the comments: Let's assume that the train will arrive at the intersection in five minutes. If you pull the lever one way, then pull it back the other, you'll save someone from losing their job. There is no chance that the lever will get stuck out that you won't be able to complete the operation on trying. Clearly pulling the lever, then pulling it back is superior to not touching it. This seems to indicate that the sin isn't pulling the lever, but pulling it without the intent to pull it back. If the sin is pulling it without intent to pull it back, then it would seem very strange that gaining the intent to pull it back, then pulling it back would be a sin.

The application of the secretary problem to real life dating

5 Elo 29 September 2015 10:28PM

The following problem is best when not described by me:


Although there are many variations, the basic problem can be stated as follows:


There is a single secretarial position to fill.

There are n applicants for the position, and the value of n is known.

The applicants, if seen altogether, can be ranked from best to worst unambiguously.

The applicants are interviewed sequentially in random order, with each order being equally likely.

Immediately after an interview, the interviewed applicant is either accepted or rejected, and the decision is irrevocable.

The decision to accept or reject an applicant can be based only on the relative ranks of the applicants interviewed so far.

The objective of the general solution is to have the highest probability of selecting the best applicant of the whole group. This is the same as maximizing the expected payoff, with payoff defined to be one for the best applicant and zero otherwise.




After reading that you can probably see the application to real life.  There are a series of bad and good assumptions following, some are fair, some are not going to be representative of you.  I am going to try to name them all as I go so that you can adapt them with better ones for yourself.  Assuming that you plan to have children and you will probably be doing so like billions of humans have done so far in a monogamous relationship while married (the entire set of assumptions does not break down for poly relationships or relationship-anarchy, but it gets more complicated).  These assumptions help us populate the Secretary problem with numbers in relation to dating for the purpose of children.


If you assume that a biological female's clock ends at 40. (in that its hard and not healthy for the baby if you try to have a kid past that age), that is effectively the end of the pure and simple biological purpose of relationships. (environment, IVF and adoption aside for a moment).  (yes there are a few more years on that)


For the purpose of this exercise – as a guy – you can add a few years for the potential age gap you would tolerate. (i.e. my parents are 7 years apart, but that seems like a big understanding and maturity gap – they don't even like the same music), I personally expect I could tolerate an age gap of 4-5 years.

If you make the assumption that you start your dating life around the ages of 16-18. that gives you about [40-18=22]  22-24 (+5 for me as a male), years of expected dating potential time.

If you estimate the number of kids you want to have, and count either:

3 years for each kid OR

2 years for each kid (+1 kid – AKA 2 years)

(Twins will throw this number off, but estimate that they take longer to recover from, or more time raising them to manageable age before you have time to have another kid)

My worked example is myself – as a child of 3, with two siblings of my own I am going to plan to have 3 children. Or 8-9 years of child-having time. If we subtract that from the number above we end up with 11-16 (16-21 for me being a male) years of dating time.

Also if you happen to know someone with a number of siblings (or children) and a family dynamic that you like; then you should consider that number of children for yourself. Remember that as a grown-up you are probably travelling through the world with your siblings beside you.  Which can be beneficial (or detrimental) as well, I would be using the known working model of yourself or the people around you to try to predict whether you will benefit or be at a disadvantage by having siblings.  As they say; You can't pick your family - for better and worse.  You can pick your friends, if you want them to be as close as a default family - that connection goes both ways - it is possible to cultivate friends that are closer than some families.  However you choose to live your life is up to you.

Assume that once you find the right person - getting married (the process of organising a wedding from the day you have the engagement rings on fingers); and falling pregnant (successfully starting a viable pregnancy) takes at least a year. Maybe two depending on how long you want to be "we just got married and we aren't having kids just yet". It looks like 9-15 (15-20 for male adjusted) years of dating.

With my 9-15 years; I estimate a good relationship of working out whether I want to marry someone, is between 6 months and 2 years, (considering as a guy I will probably be proposing and putting an engagement ring on someone's finger - I get higher say about how long this might take than my significant other does.), (This is about the time it takes to evaluate whether you should put the ring on someone's finger).  For a total of 4 serious relationships on the low and long end and 30 serious relationships on the upper end. (7-40 male adjusted relationships)

Of course that's not how real life works. Some relationships will be longer and some will be shorter. I am fairly confident that all my relationships will fall around those numbers.

I have a lucky circumstance; I have already had a few serious relationships (substitute your own numbers in here).  With my existing relationships I can estimate how long I usually spend in a relationship. (2year + 6 year + 2month + 2month /4 = 2.1 years). Which is to say that I probably have a maximum and total of around 7-15 relationships before I gotta stop expecting to have kids, or start compromising on having 3 kids.




A solution to the secretary equation

A known solution that gives you the best possible candidate the most of the time is to try out 1/e candidates (or roughly 36%), then choose the next candidate that is better than the existing candidates. For my numbers that means to go through 3-7 relationships and then choose the next relationship that is better than all the ones before.  


I don't quite like that.  It depends on how big your set is; as to what the chance of you having the best candidate in the first 1/e trials and then sticking it out till the last candidate, and settling on them.  (this strategy has a ((1/n)*(1/e)) chance of just giving you the last person in the set - which is another opportunity cost risk - what if they are rubbish? Compromise on the age gap, the number of kids or the partners quality...)  If the set is 7, the chance that the best candidate is in the first 1/e is 5.26% (if the set is 15 - the chance is much lower at 2.45%).  


Opportunity cost

Each further relationship you have might be costing you another 2 years to get further out of touch with the next generation (kids these days!)  I tend to think about how old I will be when my kids are 15-20 am I growing rapidly out of touch with the next younger generation?  Two years is a very big opportunity spend - another 2 years could see you successfully running a startup and achieving lifelong stability at the cost of the opportunity to have another kid.  I don't say this to crush you with fear of inaction; but it should factor in along with other details of your situation.


A solution to the risk of having the best candidate in your test phase; or to the risk of lost opportunity - is to lower the bar; instead of choosing the next candidate that is better than all the other candidates; choose the next candidate that is better than 90% of the candidates so far.  Incidentally this probably happens in real life quite often.  In a stroke of, "you'll do"...


Where it breaks down


Real life is more complicated than that. I would like to think that subsequent relationships that I get into will already not suffer the stupid mistakes of the last ones; As well as the potential opportunity cost of exploration. The more time you spend looking for different partners – you might lose your early soul mate, or might waste time looking for a better one when you can follow a "good enough" policy. No one likes to know they are "good enough", but we do race the clock in our lifetimes. Life is what happens when you are busy making plans.


As someone with experience will know - we probably test and rule out bad partners in a single conversation, where we don't even get so far as a date.  Or don't last more than a week. (I. E the experience set is growing through various means).


People have a tendency to overrate the quality of a relationship while they are in it, versus the ones that already failed.


Did I do something wrong? 

“I got married early - did I do something wrong (or irrational)?”

No.  equations are not real life.  It might have been nice to have the equation, but you obviously didn't need it.  Also this equation assumes a monogamous relationship.  In real life people have overlapping relationships, you can date a few people and you can be poly. These are all factors that can change the simple assumptions of the equation. 


Where does the equation stop working?

Real life is hard.  It doesn't fall neatly into line, it’s complicated, it’s ugly, it’s rough and smooth and clunky.  But people still get by.  Don’t be afraid to break the rule. 

Disclaimer: If this equation is the only thing you are using to evaluate a relationship - it’s not going to go very well for you.  I consider this and many other techniques as part of my toolbox for evaluating decisions.

Should I break up with my partner?

What? no!  Following an equation is not a good reason to live your life.  

Does your partner make you miserable?  Then yes you should break up.


Do you feel like they are not ready to have kids yet and you want to settle down?  Tough call.  Even if they were agents also doing the equation; An equation is not real life.  Go by your brain; go by your gut.  Don’t go by just one equation.

Expect another post soon about reasonable considerations that should be made when evaluating relationships.

The given problem makes the assumption that you are able to evaluate partners in the sense that the secretary problem expects.  Humans are not all strategic and can’t really do that.  This is why the world is not going to perfectly follow this equation.  Life is complicated; there are several metrics that make a good partner and they don’t always trade off between one another.



Meta: writing time - 3 hours over a week; 5+ conversations with people about the idea, bothering a handful of programmers and mathematicians for commentary on my thoughts, and generally a whole bunch of fun talking about it.  This post was started on the slack channel when someone asked a related question.


My table of contents for other posts in my series.


Let me know if this post was helpful or if it worked for you or why not.

Ultimatums in the Territory

10 malcolmocean 28 September 2015 10:01PM

When you think of "ultimatums", what comes to mind?

Manipulativeness, maybe? Ultimatums are typically considered a negotiation tactic, and not a very pleasant one.

But there's a different thing that can happen, where an ultimatum is made, but where articulating it isn't a speech act but rather an observation. As in, the ultimatum wasn't created by the act of stating it, but rather, it already existed in some sense.

Some concrete examples: negotiating relationships

I had a tense relationship conversation a few years ago. We'd planned to spend the day together in the park, and I was clearly angsty, so my partner asked me what was going on. I didn't have a good handle on it, but I tried to explain what was uncomfortable for me about the relationship, and how I was confused about what I wanted. After maybe 10 minutes of this, she said, "Look, we've had this conversation before. I don't want to have it again. If we're going to do this relationship, I need you to promise we won't have this conversation again."

I thought about it. I spent a few moments simulating the next months of our relationship. I realized that I totally expected this to come up again, and again. Earlier on, when we'd had the conversation the first time, I hadn't been sure. But it was now pretty clear that I'd have to suppress important parts of myself if I was to keep from having this conversation.

"...yeah, I can't promise that," I said.

"I guess that's it then."

"I guess so."

I think a more self-aware version of me could have recognized, without her prompting, that my discomfort represented an unreconcilable part of the relationship, and that I basically already wanted to break up.

The rest of the day was a bit weird, but it was at least nice that we had resolved this. We'd realized that it was a fact about the world that there wasn't a serious relationship that we could have that we both wanted.

I sensed that when she posed the ultimatum, she wasn't doing it to manipulate me. She was just stating what kind of relationship she was interested in. It's like if you go to a restaurant and try to order a pad thai, and the waiter responds, "We don't have rice noodles or peanut sauce. You either eat somewhere else, or you eat something other than a pad thai."

An even simpler example would be that at the start of one of my relationships, my partner wanted to be monogamous and I wanted to be polyamorous (i.e. I wanted us both to be able to see other people and have other partners). This felt a bit tug-of-war-like, but eventually I realized that actually I would prefer to be single than be in a monogamous relationship.

I expressed this.

It was an ultimatum! "Either you date me polyamorously or not at all." But it wasn't me "just trying to get my way".

I guess the thing about ultimatums in the territory is that there's no bluff to call.

It happened in this case that my partner turned out to be really well-suited for polyamory, and so this worked out really well. We'd decided that if she got uncomfortable with anything, we'd talk about it, and see what made sense. For the most part, there weren't issues, and when there were, the openness of our relationship ended up just being a place where other discomforts were felt, not a generator of disconnection.

Normal ultimatums vs ultimatums in the territory

I use "in the territory" to indicate that this ultimatum isn't just a thing that's said but a thing that is true independently of anything being said. It's a bit of a poetic reference to the map-territory distinction.

No bluffing: preferences are clear

The key distinguishing piece with UITTs is, as I mentioned above, that there's no bluff to call: the ultimatum-maker isn't secretly really really hoping that the other person will choose one option or the other. These are the two best options as far as they can tell. They might have a preference: in the second story above, I preferred a polyamorous relationship to no relationship. But I preferred both of those to a monogamous relationship, and the ultimatum in the territory was me realizing and stating that.

This can actually be expressed formally, using what's called a preference vector. This comes from Keith Hipel at University of Waterloo. If the tables in this next bit doesn't make sense, don't worry about it: all important conclusions are expressed in the text.

First, we'll note that since each of us have two options, a table can be constructed which shows four possible states (numbered 0-3 in the boxes).

    My options
  options insist poly don't insist
offer relationship 3: poly relationship 1: mono relationship
don't offer 2: no relationship 0: (??) no relationship

This representation is sometimes referred to as matrix form or normal form, and has the advantage of making it really clear who controls which state transitions (movements between boxes). Here, my decision controls which column we're in, and my partner's decision controls which row we're in.

Next, we can consider: of these four possible states, which are most and least preferred, by each person? Here's my preferences, ordered from most to least preferred, left to right. The 1s in the boxes mean that the statement on the left is true.

state 3 2 1 0
I insist on polyamory 1 1 0 0
partner offers relationship 1 0 1 0
My preference vector (← preferred)

The order of the states represents my preferences (as I understand them) regardless of what my potential partner's preferences are. I only control movement in the top row (do I insist on polyamory or not). It's possible that they prefer no relationship to a poly relationship, in which case we'll end up in state 2. But I still prefer this state over state 1 (mono relationship) and state 0 (in which I don't ask for polyamory and my partner decides not to date me anyway). So whatever my partners preferences are, I've definitely made a good choice for me, by insisting on polyamory.

This wouldn't be true if I were bluffing (if I preferred state 1 to state 2 but insisted on polyamory anyway). If I preferred 1 to 2, but I bluffed by insisting on polyamory, I would basically be betting on my partner preferring polyamory to no relationship, but this might backfire and get me a no relationship, when both of us (in this hypothetical) would have preferred a monogamous relationship to that. I think this phenomenon is one reason people dislike bluffy ultimatums.

My partner's preferences turned out to be...

state 1 3 2 0
I insist on polyamory 0 1 1 0
partner offers relationship 1 1 0 0
Partner's preference vector (← preferred)

You'll note that they preferred a poly relationship to no relationship, so that's what we got! Although as I said, we didn't assume that everything would go smoothly. We agreed that if this became uncomfortable for my partner, then they would tell me and we'd figure out what to do. Another way to think about this is that after some amount of relating, my partner's preference vector might actually shift such that they preferred no relationship to our polyamorous one. In which case it would no longer make sense for us to be together.

UITTs release tension, rather than creating it

In writing this post, I skimmed a wikihow article about how to give an ultimatum, in which they say:

"Expect a negative reaction. Hardly anyone likes being given an ultimatum. Sometimes it may be just what the listener needs but that doesn't make it any easier to hear."

I don't know how accurate the above is in general. I think they're talking about ultimatums like "either you quit smoking or we break up". I can say that expect that these properties of an ultimatum contribute to the negative reaction:

  • stated angrily or otherwise demandingly
  • more extreme than your actual preferences, because you're bluffing
  • refers to what they need to do, versus your own preferences

So this already sounds like UITTs would have less of a negative reaction.

But I think the biggest reason is that they represent a really clear articulation of what one party wants, which makes it much simpler for the other party to decide what they want to do. Ultimatums in the territory tend to also be more of a realization that you then share, versus a deliberate strategy. And this realization causes a noticeable release of tension in the realizer too.

Let's contrast:

"Either you quit smoking or we break up!"


"I'm realizing that as much as I like our relationship, it's really not working for me to be dating a smoker, so I've decided I'm not going to. Of course, my preferred outcome is that you stop smoking, not that we break up, but I realize that might not make sense for you at this point."

Of course, what's said here doesn't necessarily correspond to the preference vectors shown above. Someone could say the demanding first thing when they actually do have a UITT preference-wise, and someone who's trying to be really NVCy or something might say the sceond thing even though they're actually bluffing and would prefer to . But I think that in general they'll correlate pretty well.

The "realizing" seems similar to what happened to me 2 years ago on my own, when I realized that the territory was issuing me an ultimatum: either you change your habits or you fail at your goals. This is how the world works: your current habits will get you X, and you're declaring you want Y. On one level, it was sad to realize this, because I wanted to both eat lots of chocolate and to have a sixpack. Now this ultimatum is really in the territory.

Another example could be realizing that not only is your job not really working for you, but that it's already not-working to the extent that you aren't even really able to be fully productive. So you don't even have the option of just working a bit longer, because things are only going to get worse at this point. Once you realize that, it can be something of a relief, because you know that even if it's hard, you're going to find something better than your current situation.

Loose ends

More thoughts on the break-up story

One exercise I have left to the reader is creating the preference vectors for the break-up in the first story. HINT: (rot13'd) Vg'f fvzvyne gb gur cersrerapr irpgbef V qvq fubj, jvgu gjb qrpvfvbaf: fur pbhyq vafvfg ba ab shgher fhpu natfgl pbairefngvbaf be abg, naq V pbhyq pbagvahr gur eryngvbafuvc be abg.

An interesting note is that to some extent in that case I wasn't even expressing a preference but merely a prediction that my future self would continue to have this angst if it showed up in the relationship. So this is even more in the territory, in some senses. In my model of the territory, of course, but yeah. You can also think of this sort of as an unconscious ultimatum issued by the part of me that already knew I wanted to break up. It said "it's preferable for me to express angst in this relationship than to have it be angst free. I'd rather have that angst and have it cause a breakup than not have the angst."

Revealing preferences

I think that ultimatums in the territory are also connected to what I've called Reveal Culture (closely related to Tell Culture, but framed differently). Reveal cultures have the assumption that in some fundamental sense we're on the same side, which makes negotiations a very different thing... more of a collaborative design process. So it's very compatible with the idea that you might just clearly articulate your preferences.

Note that there doesn't always exist a UITT to express. In the polyamory example above, if I'd preferred a mono relationship to no relationship, then I would have had no UITT (though I could have bluffed). In this case, it would be much harder for me to express my preferences, because if I leave them unclear then there can be kind of implicit bluffing. And even once articulated, there's still no obvious choice. I prefer this, you prefer that. We need to compromise or something. It does seem clear that, with these preferences, if we don't end up with some relationship at the end, we messed up... but deciding how to resolve it is outside the scope of this post.

Knowing your own preferences is hard

Another topic this post will point at but not explore is: how do you actually figure out what you want? I think this is a mix of skill and process. You can get better at the general skill by practising trying to figure it out (and expressing it / acting on it when you do, and seeing if that works out well). One process I can think of that would be helpful is Gendlin's Focusing. Nate Soares has written about how introspection is hard and to some extent you don't ever actually know what you want: You don't get to know what you're fighting for. But, he notes,

"There are facts about what we care about, but they aren't facts about the stars. They are facts about us."

And they're hard to figure out. But to the extent that we can do so and then act on what we learn, we can get more of what we want, in relationships, in our personal lives, in our careers, and in the world.

(This article crossposted from my personal blog.)

Examples of growth mindset or practice in fiction

11 Swimmer963 28 September 2015 09:47PM

As people who care about rationality and winning, it's pretty important to care about training. Repeated practice is how humans acquire skills, and skills are what we use for winning.

Unfortunately, it's sometimes hard to get System 1 fully on board with the fact that repeated, difficult, sometimes tedious practice is how we become awesome. I find fiction to be one of the most useful ways of communicating things like this to my S1. It would be great to have a repository of fiction that shows characters practicing skills, mastering them, and becoming awesome, to help this really sink in.

However, in fiction the following tropes are a lot more common:

  1. hero is born to greatness and only needs to discover that greatness to win [I don't think I actually need to give examples of this?]
  2. like (1), only the author talks about the skill development or the work in passing… but in a way that leaves the reader's attention (and system 1 reinforcement?) on the "already be awesome" part, rather that the "practice to become awesome" part [HPMOR; the Dresden Files, where most of the implied practice takes place between books.]
  3. training montage, where again the reader's attention isn't on the training long enough to reinforce the "practice to become awesome" part, but skips to the "wouldn't it be great to already be awesome" part [TVtropes examples].
  4. The hero starts out ineffectual and becomes great over the course of the book, but this comes from personal revelations and insights, rather than sitting down and practicing [Nice Dragons Finish Last is an example of this].

Example of exactly the wrong thing:
The Hunger Games - Katniss is explicitly up against the Pledges who have trained their whole lives for this one thing, but she has … something special that causes her to win. Also archery is her greatest skill, and she's already awesome at it from the beginning of the story and never spends time practicing.

Close-but-not-perfect examples of the right thing:
The Pillars of the Earth - Jack pretty explicitly has to travel around Europe to acquire the skills he needs to become great. Much of the practice is off-screen, but it's at least a pretty significant part of the journey.
The Honor Harrington series: the books depict Honor, as well as the people around her, rising through the ranks of the military and gradually levelling up, with emphasis on dedication to training, and that training is often depicted onscreen – but the skills she's training in herself and her subordinates aren't nearly as relevant as the "tactical genius" that she seems to have been born with.

I'd like to put out a request for fiction that has this quality. I'll also take examples of fiction that fails badly at this quality, to add to the list of examples, or of TVTropes keywords that would be useful to mine. Internet hivemind, help?

Open thread, Sep. 28 - Oct. 4, 2015

3 MrMind 28 September 2015 07:13AM

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

Dry Ice Cryonics- Preliminary Thoughts

8 Fluttershy 28 September 2015 07:00AM

This post is a spot-check of Alcor's claim that cryonics can't be carried out at dry ice temperatures, and a follow-up to this comment. This article isn't up to my standards, yet I'm posting it now, rather than polishing it more first, because I strongly fear that I might never get around to doing so later if I put it off. Despite my expertise in chemistry, I don't like chemistry, so writing this took a lot of willpower. Thanks to Hugh Hixon from Alcor for writing "How Cold is Cold Enough?".


More research (such as potentially hiring someone to find the energies of activation for lots of different degradative reactions which happen after death) is needed to determine if long-term cryopreservation at the temperature of dry ice is reasonable, or even preferable to storage in liquid nitrogen.

On the outside view, I'm not very confident that dry ice cryonics will end up being superior to liquid nitrogen cryonics. Still, it's very hard to say one way or the other a priori. There are certain factors that I can't easily quantify that suggest that cryopreservation with dry ice might be preferable to cryopreservation with liquid nitrogen (specifically, fracturing, as well as the fact that the Arrhenius equation doesn't account for poor stirring), and other such factors that suggest preservation in liquid nitrogen to be preferable (specifically, that being below the glass transition temperature prevents movement/chemical reactions, and that nanoscale ice crystals, which can grow during rewarming, can form around the glass transition temperature).

(I wonder if cryoprotectant solutions with different glass transition temperatures might avoid either of the two problems mentioned in the last sentence for dry ice cryonics? I just heard about the issue of nanoscale ice crystals earlier today, so my discussion of them is an afterthought.)


Using dry ice to cryopreserve people for future revival could be cheaper than using liquid nitrogen for the same purpose (how much would using dry ice cost?). Additionally, lowering the cost of cryonics could increase the number of people who sign up for cryonics-- which would, in turn, give us a better chance at e.g. legalizing the initiation of the first phases of cryonics for terminal patients just before legal death.

This document by Alcor suggests that, for neuro and whole-body patients, an initial deposit of 6,600 or 85,438 USD into the patient's trust fund is, respectively, more than enough to generate enough interest to safely cover a patient's annual storage cost indefinitely. Since around 36% of this amount is spent on liquid nitrogen, this means that completely eliminating the cost of replenishing the liquid nitrogen in the dewars would reduce the up-front cost that neuro and whole-body patients with Alcor would pay by around 2,350 or 31,850 USD, respectively. This puts a firm upper bound on the amount that could be saved by Alcor patients by switching to cryopreservation with dry ice, since some amount would need to be spent each year on purchasing additional dry ice to maintain the temperature at which patients are stored. (A small amount could probably be saved on the cost which comes from cooling patients down immediately after death, as well).

This LW discussion is also relevant to storage costs in cryonics. I'm not sure how much CI spends on storage.

Relevant Equations and Their Limitations

Alcor's "How Cold is Cold Enough?" is the only article which I've found that takes an in-depth look at whether storage of cryonics patients at temperatures above the boiling point of liquid nitrogen would be feasible. It's a generally well-written article, though it makes an assumption regarding activation energy that I'll be forced to examine later on.

The article starts off by introducing the Arrhenius equation, which is used to determine the rate constant of a chemical reaction at a given temperature. The equation is written:

k = A * e^(-Ea/RT)          (1)


  • k is the rate constant you solve for (the units vary between reactions)
  • A is a constant you know (same units as k)
  • Ea is the activation energy (kJ/mol)
  • R is the ideal gas constant (kJ/K*mol)
  • T is the temperature (K)
As somewhat of an aside, this is the same k that you would plug into rate law equation, which you have probably seen before:

v = k * [A]m[B]n     (2)
  • v is the rate of the reaction (mol/(L*s))
  • k is the rate constant, from the Arrhenius equation above
  • [A] and [B] are the concentrations of reactants-- there might be more or less than two (mol/L)
  • m and n are constants that you know
The Arrhenius equation-- equation 1, here-- does make some assumptions which don't always hold. Firstly, the activation energy of some reactions changes with temperature, and secondly, it is sometimes necessary to use the modified Arrhenius equation (not shown here) to fit rate constant v. temperature data, as noted just before equation 5 in this paper. This is worth mentioning because, while the Arrhenius equation is quite robust, the data doesn't always fit our best models in chemistry.

Lastly, and most importantly, the Arrhenius equation assumes that all reactants are always being mixed perfectly, which is definitely not the case in cryopreserved patients. I have no idea how to quantify this effect, though after taking this effect into consideration, we should expect degradation reactions in cryopreserved individuals to happen much more slowly than the Arrhenius equation would explicitly predict.

Alcor on "How Cold is Cold Enough?"

The Alcor article goes on to calculate the ratio of the value of k, the rate constant, at 77.36 Kelvin (liquid nitrogen), to the value of k at other temperatures for the enzyme Catalase. This ratio is equal to the factor by which a reaction would be slowed down when cooled from a given temperature down to 77 K. While the calculations are correct, Catalase is not the ideal choice of enzyme here. Ideally, we'd want to calculate this ratio for whatever degradative enzyme/reaction had the lowest activation energy, because then, if the ratio of k at 37 Celsius (body temperature) to k at the temperature of dry ice was big enough, we could be rather confident that all other degradative reactions would be slowed down at dry ice temperatures by a greater factor than the degradative reaction with the lowest activation energy would be. Of course, as shown in equation 2 of this post, the concentrations of reactants of degradative reactions do matter to the speed of those reactions at dry ice temperatures, though differences in the ratio of k at 37 C to k at dry ice temperatures between different degradative reactions will matter much, much more strongly in determining v, the rate of the reaction, than differences in concentrations of reactants will.

I'm also quite confused by the actual value given for the Ea of catalase in the Alcor article-- a quick google search suggests the Ea to be around 8 kJ/mol or 11 kJ/mol, though the Alcor article uses a value of 7,000 cal/(mol*K), i.e. 29.3 kJ/(mol*K), which can only be assumed to have been a typo in terms of the units used.

Of course, as the author mentions, Ea values aren't normally tabulated. The Ea for a reaction can be calculated with just two experimentally determined (Temperature, k (rate constant)) pairs, so it wouldn't take too long to experimentally determine a bunch of Eas for degradative reactions which normally take place in the human body after death, especially if we could find a biologist who had a good a priori idea of which degradative reactions would be the fastest.

Using the modified form of the Arrhenius equation from Alcor's "How Cold is Cold Enough", we could quickly estimate what the smallest Ea for a degradative biological reaction would be that would result in some particular and sufficiently small number of reactions taking place at dry ice temperatures over a certain duration of time. For example, when neglecting stirring effects, it turns out that 100 years at dry ice temperature (-78.5 C) ought to be about equal to 3 minutes at body temperature for a reaction with an Ea of 72.5 kJ/mol. Reactions with higher Eas would be slowed down relatively more by an identical drop in temperature.

So, if we were unable to find any degradative biological reactions with Eas less than (say) 72.5 kJ/mol, that would be decent evidence in favor of dry ice cryonics working reasonably well (given that the 100 years and three minutes figures are numbers that I just made up-- 100 years being a possible duration of storage, and three minutes being an approximation of how long one can live without oxygen being supplied to the brain).

Damage from Causes Other Than Chemical Reactions in Dry Ice Cryonics

Just before publishing this article, I came across Alcor's "Cryopreservation and Fracturing", which mentioned that 

The most important instability for cryopreservation purposes is a tendency toward ice nucleation. At temperatures down to 20 degrees below the glass transition temperature, water molecules are capable of small translations and rotations to form nanoscale ice-crystals, and there is strong thermodynamic incentive to do so [5, 6]. These nanoscale crystals (called "nuclei") remain small and biologically insignificant below the glass transition, but grow quickly into damaging ice crystals as the temperature rises past -90°C during rewarming. Accumulating ice nuclei are therefore a growing liability that makes future ice-free rewarming efforts progressively more difficult the longer vitrified tissue is stored near the glass transition temperature. For example, storing a vitrification solution 10 degrees below the glass transition for six months was found to double the warming rate necessary to avoid ice growth during rewarming [5]. The vitrification solution that Alcor uses is far more stable than the solution used (VS41A) in this particular experiment, but Alcor must store its patients far longer than six months.

The same article also discusses fracturing, which can damage tissues stored more than 20 C below the glass transition temperature. If nanoscale ice crystals form in patients stored in dry ice (I expect they would), and grew during rewarming from dry ice temperatures (I have no idea if they would), that could be very problematic.

Implications of this Research for Liquid Nitrogen Cryonics

If someone has a graph of how body temperature varies with time during the process of cryopreservation, it would be trivial to compute the time-at-body-temperature equivalent of the time that freezing takes. My bet is that getting people frozen too slowly hurts folks's chances of revival far more than they intuit.

Instrumental Rationality Questions Thread

6 AspiringRationalist 27 September 2015 09:22PM

Previous thread: http://lesswrong.com/lw/mnq/instrumental_rationality_questions_thread/

This thread is for asking the rationalist community for practical advice.  It's inspired by the stupid questions series, but with an explicit focus on instrumental rationality.

Questions ranging from easy ("this is probably trivial for half the people on this site") to hard ("maybe someone here has a good answer, but probably not") are welcome.  However, please stick to problems that you actually face or anticipate facing soon, not hypotheticals.

As with the stupid questions thread, don't be shy, everyone has holes in their knowledge, though the fewer and the smaller we can make them, the better, and please be respectful of other people's admitting ignorance and don't mock them for it, as they're doing a noble thing.

(See also the Boring Advice Repository)

Hypothetical situations are not meant to exist

1 casebash 27 September 2015 10:58AM

Hypotheticals are a powerful tool for testing intuitions. However, many people believe that it is problematic a hypothetical does not represent a realistic situation. On the contrary, it is only problematic if it is represented as being realistic when it is not realistic. Realism isn’t required if the aim is simply to show that there is *some* situation where the proposed principle breaks. We may still choose to utilise an imperfect principle, but when we know about the potential for breakage, we are much less likely to be tripped up if we find a situation where the principle is invalid.

It is instructive to look at physics. In physics, we model balls by perfect spherical objects. Nobody believes that a perfectly spherical object exists in real life. However, they provide a baseline theory from which further ideas can be explored. Bumps or ellipticity can be added later. Indeed, they probably *should* be added later. Unless a budding physicist can demonstrate their competence with the simple case, they probably should not be trusted with dealing with the much more complicated real world situation.

If you are doubting a hypothetical, then you haven’t accepted the hypothetical. You can doubt that a hypothetical will have any relevance from outside the hypothetical, but once you step inside the hypothetical you cannot doubt the hypothetical or you never stepped inside in the first place.

This topic has been discussed previously on LessWrong, but a single explanation won't prove compelling to everyone, so it is useful to have different explanations that explain the same topic in a different way.

TimS states similar thoughts in Please Don’t Fight the Hypothetical:

Likewise, people who responds to the Trolley problem by saying that they would call the police are not talking about the moral intuitions that the Trolley problem intends to explore.  There's nothing wrong with you if those problems are not interesting to you.  But fighting the hypothetical by challenging the premises of the scenario is exactly the same as saying, "I don't find this topic interesting for whatever reason, and wish to talk about something I am interested in."

In, The Least Convenient World, Yvain recommends limiting your responses as follows:

[Say] "I completely reject the entire basis of your argument" or "I accept the basis of your argument, but it doesn't apply to the real world because of contingent fact X." If you just say "Yeah, well, contigent fact X!" and walk away, you've left yourself too much wiggle room.

You may also want to check out A note on hypotheticals by PhilGoetz



Find someone to talk to thread

22 hg00 26 September 2015 10:24PM

Many LessWrong users are depressed. On the most recent survey, 18.2% of respondents had been formally diagnosed with depression, and a further 25.5% self-diagnosed with depression. That adds up to nearly half of the LessWrong userbase.

One common treatment for depression is talk therapy. Jonah Sinick writes:

Talk therapy has been shown to reduce depression on average. However:

  • Professional therapists are expensive, often charging on order of $120/week if one's insurance doesn't cover them.
  • Anecdotally, highly intelligent people find therapy less useful than the average person does, perhaps because there's a gap in intelligence between them and most therapists that makes it difficult for the therapist to understand them.

House of Cards by Robyn Dawes argues that there's no evidence that licensed therapists are better at performing therapy than minimally trained laypeople. The evidence therein raises the possibility that one can derive the benefits of seeing a therapist from talking to a friend.

This requires that one has a friend who:

  • is willing to talk with you about your emotions on a regular basis
  • you trust to the point of feeling comfortable sharing your emotions

Some reasons to think that talking with a friend may not carry the full benefits of talking with a therapist are

  • Conflict of interest — Your friend may be biased for reasons having to do with your pre-existing relationship – for example, he or she might be unwilling to ask certain questions or offer certain feedback out of concern of offending you and damaging your friendship.
  • Risk of damaged relationship dynamics — There's a possibility of your friend feeling burdened by a sense of obligation to help you, creating feelings of resentment, and/or of you feeling guilty.
  • Risk of breach of confidentiality — Since you and your friend know people in common, there's a possibility that your friend will reveal things that you say to others who you know, that you might not want to be known. In contrast, a therapist generally won't know people in common with you, and is professionally obliged to keep what you say confidential.

Depending on the friend and on the nature of help that you need, these factors may be non-issues, but they're worth considering when deciding between seeing a therapist and talking with a friend.

One idea for solving the problems with talking to a friend is to find someone intellectually similar to you who you don't know--say, someone else who reads LessWrong.

This is a thread for doing that. Please post if you're either interested in using someone as a sounding board or interested in making money being a sounding board using Skype or Google Hangouts. If you want to make money talking to people, I suggest writing out a little resume describing why you might be a nice person to talk to, the time zone you're in, your age (age-matching recommended by Kate), and the hourly rate you wish to charge. You could include your location for improved internet call quality. You might also include contact info to decrease trivial inconveniences for readers who haven't registered a LW account. (I have a feeling that trivial inconveniences are a bigger issue for depressed people.) To help prevent email address harvesting, the convention for this thread is if you write "Contact me at [somename]", that's assumed to mean "my email is [somename]@gmail.com".

Please don't be shy about posting if this sounds like a good fit for you. Let's give people as many options as possible.

I guess another option for folks on a budget is making reciprocal conversation arrangements with another depressed person. So feel free to try & arrange that in this thread as well. I think paying someone is ideal though; listening to depressed people can sometimes be depressing.

BlahTherapy is an interesting site that sets you up with strangers on the internet to talk about your problems with. However, these strangers likely won't have the advantages of high intelligence or shared conceptual vocabulary LessWrong users have. Fortunately we can roll our own version of BlahTherapy by designating "lesswrong-talk-to-someone" as the Schelling interest on Omegle.com. (You can also just use lesswrong as an interest, there are sometimes people on. Or enter random intellectual interests to find smart people to talk to.)

I haven't had very good results using sites like BlahTherapy. I think it's because I only sometimes find someone good, and when they don't work, I end up more depressed than I started. Reaching out in hopes of finding a friend and failing is a depressing experience. So I recommend trying to create a stable relationship with regularly scheduled conversations. I included BlahTherapy and Omegle because they might work well for some people and I didn't want to extrapolate strongly from n=1.

LessWrong user ShannonFriedman seems to work as a life coach judging by the link in her profile. I recommend her posts How to Deal with Depression - The Meta Layers and The Anti-Placebo Effect.

There's also the How to Get Therapy series from LW-sphere blog Gruntled & Hinged. It's primarily directed at people looking for licensed therapists, but may also have useful tips if you're just looking for someone to talk to. The biggest tip I noticed was to schedule a relaxing activity & time to decompress after your conversation.

The book Focusing is supposed to explain the techniques that successful therapy patients use that separate them from unsuccessful therapy patients.  Anna Salamon recommends the audiobook version.

There's also: Methods for Treating DepressionThings That Sometimes Help If You Have Depression.

I apologize for including so many ideas, but I figured it was better to suggest a variety of approaches so the community can collectively identify the most effective solutions for the rationalist depression epidemic. In general, when I'm depressed, I notice myself starting and stopping activities in a very haphazard way, repeatedly telling myself that the activity I'm doing isn't the one I "should" be doing. I've found it pretty useful to choose one activity arbitrarily and persist in it for a while. This is often sufficient to bootstrap myself out of a depressed state. I'd recommend doing the same here: choose an option and put a nontrivial amount of effort into exploring it before discarding it. Create a todo list and bulldoze your way down it.

Good luck. I'm rooting for you!

Legal note: Talking to unlicensed people over the internet is not a substitute for professional help. If you are depressed you should visit a licensed therapist.

[Link] 2015 modafinil user survey

9 gwern 26 September 2015 05:28PM

I am running, in collaboration with ModafinilCat, a survey of modafinil users asking about their experiences, side-effects, sourcing, efficacy, and demographics:

This is something of a followup to the LW surveys which find substantial modafinil use, and Yvain's 2014 nootropics survey. I hope the results will be useful; the legal questions should help reduce uncertainty there, and the genetics questions (assuming any responses) may be interesting too.

Happy Petrov Day

9 Eneasz 26 September 2015 03:41PM

It is Petrov Day again, partially thanks to Stanislaw Petrov.

"Today is September 26th, Petrov Day, celebrated to honor the deed of Stanislav Yevgrafovich Petrov on September 26th, 1983.  Wherever you are, whatever you're doing, take a minute to not destroy the world."


[Link] Review of "Doing Good Better"

0 fortyeridania 26 September 2015 07:58AM

The article is here.

The book is by William MacAskill, founder of 80000 Hours and Giving What We Can. Excerpt:

Effective altruism takes up the spirit of Singer’s argument but shields us from the full blast of its conclusion; moral indictment is transformed into an empowering investment opportunity...

Either effective altruism, like utilitarianism, demands that we do the most good possible, or it asks merely that we try to make things better. The first thought is genuinely radical, requiring us to overhaul our daily lives in ways unimaginable to most...The second thought – that we try to make things better – is shared by every plausible moral system and every decent person. If effective altruism is simply in the business of getting us to be more effective when we try to help others, then it’s hard to object to it. But in that case it’s also hard to see what it’s offering in the way of fresh moral insight, still less how it could be the last social movement we’ll ever need.

Help with relevant rational decision making research ideas

1 BiasedBayes 25 September 2015 05:46PM

Im researching rational medical decision making and trying to find right angles to approach the subject of probabilistic decision making in medicine.

My first paper will be about the probability updating after the medical tests when we know the prevalence of the patients possible disease and the tests sensitivity and specificity.
Second paper will be (at the moment) about the decision to test if we have certain treatment treshold and know the prevalence of the disease and the tests sensitivity and specifity.

So im reasearching how well the descriptive medical decision making (usually intuitive) matches the normative ideal when in the decision making situation we have all the available info to actually calculate the best action.

If you have some ideas regarding this topic I would love to hear them. Thanks a lot!

New LW Meetup: Lund

1 FrankAdamek 25 September 2015 03:40PM

Subjective vs. normative offensiveness

2 casebash 25 September 2015 04:10AM

Terms like offensive are often used in a manner that blurs the boundaries between two different, but related concepts. Let’s suppose that Alex send Billy an email that he finds offensive. We can say that the email is subjectively offensive if it causes Billy to feel offended. On the other hand, we can say that it is normatively offensive if Alex has taken an action that deserves to be criticised morally. The former does not prove the later. For example, Alex might tell Billy that he saw a recent Mets game and that he thought their pitcher didn’t play very well. Billy might be a huge Mets fan and find this offensive. Clearly the Alex was subjectively offensive (relative to Billy), but few people would say that he was normatively offensive. This requires something extra, such as if Alex had made a similar comment before and seen that it had upset Billy we might be more willing to conclude that Alex deserved the criticism.

Billy is entitled to feel an emotional reaction and feel offended (subjectively). It would be hard to argue that he isn’t as can be incredibly difficult or impossible to suppress such a reaction. However, he is not entitled to act like Alex was normatively offensive purely based on his subjective appraisal. He needs to consider the actual reasonableness of Alex’s actions and the broader social context. Sadly, this normally results in very messy conversations. One side will be arguing, “You shouldn’t be (normatively) offended”, with the other saying that they have every right to be (subjectively) offended.

At this point, I should clarify the greatest misunderstanding based upon feedback in the comments. Normative here simply refers to some kind of moral standard; to the making of claims that people should act in a particular way. It doesn't depend on the assumption that morality is objective; just that the person operates within some kind of moral framework that leaves their moral assertions open to challenge by others. In regard to culturally relativism, normative is being used to mean "locally normative"; normative within the particular cultural context; rather than "globally normative". Even if you believe that morality is purely personal, you probably have some meta-level beliefs that can be used to challenge or justify your object level beliefs.

I think that this discussion is well understood for the word offensive, but it is less well understood for the term creepy. Suppose there is a man who has unfortunately born to be incredibly ugly. People may find it subjectively creepy for him to just walk into the room. However, it isn’t normatively creepy for him to enter the room; you can’t criticise him merely for the action.

This is enough to resolve many discussions about creepiness. Someone’s age when they age you out or someone being unattractive may make them subjectively creepy. This is an emotional response and we can’t expect people to suppress all of their emotional responses. However, it requires extra work to prove that something is normatively creepy. In particular, there needs to be a reference to some kind of moral rule or social contract. Like if a 20 year old wanted to criticise a 40 year old for being normatively creepy for asking them out, they’d have to argue that they have broken some social or moral rule like, “Don’t ask someone out half your age”. They would then have to justify this rule; arguing purely from their own subjective point of view would be insufficient.

Many people will say that when they said that something was offensive that they only meant it was offensive from their own perspective. However, people are almost forced to respond as though it was meant in the normative sense. These words are used in a social environment and any such comment carries a significant chance that people will judge them as guilty in some normative sense, even if that wasn’t what you meant. If you really want to mean it in the subjective sense, then you need to add padding around it or it will be misunderstood. Scott Alexander wrote a good article on this about Weak Men, but I won’t link directly to it as it is likely to be very controversial.

Lastly, this relates strongly to Motte and Bailey doctrine. It can often be very convenient for someone to say that an action is offensive (with the assumption being that it is offensive in the normative sense) and then fall back down to saying that they were only personally offended when the other person tries to defend them self, then start talking about being offended in the normative sense again, right after. Please don’t do this.

The purpose of this article wasn’t just to draw this distinction, but to also provide terminology to make these distinctions as clear as possible. Please try to avoid argumentation in the comments about the actual object level issues as I have tried to avoid directly tackling the issue of who is or isn’t correct and to keep it at the level of how to generally approach these issues.

What is your rationalist backstory?

7 adamzerner 25 September 2015 01:25AM

I'm reading Dan Ariely's book Predictably Irrational. The story of what got him interested in rationality and human biases goes something like this.

He was the victim of a really bad accident, and had terrible burns covering ~70% of his body. The experience was incredibly painful, and so was the treatment. For treatment, he'd have to bathe in some sort of disinfectant, and then have bandages ripped off his exposed flesh afterwards, which was extremely painful for him.

The nurses believed that ripping it off quickly would produce the least amount of pain for the patient. They thought the short and intense bursts of pain were less (in aggregate) than the less intense but longer periods of pain that a slower removal of the bandages would produce. However, Dan disagreed about what would produce the least amount of pain for patients. He thought that a slower removal would be better. Eventually, he found some scientific research that supported/proved his theory to be correct.

But he was confused. These nurses were smart people and had a ton of experience giving burn victims baths - shouldn't they have figured out by now what approaches best minimize patient pain? He knew their failure wasn't due to a lack of intelligence, and that it wasn't due to a lack of sympathy. He ultimately concluded that the failure was due to inherent human biases. He then became incredibly interested in this and went on to do a bunch of fantastic research in the area.

In my experience, the overwhelming majority of people are uninterested in rationality, and a lot of them are even put off by it. So I'm curious about how members of this incredibly small minority of the population became who they are.

Part of me thinks that extreme outputs are the result of extreme inputs. Like how Dan's extreme passion for his work has (seemingly) originated from his extreme experiences with pain. With this rule-of-thumb in mind, when I see someone who possesses some extreme character trait, I expect there to be some sort of extreme story or experience behind it.

But another part of me thinks that this doesn't really apply to rationality. I don't have much data, but from the limited experience I've had getting to know people in this community, "I've just always thought this way" seems common, and "extreme experiences that motivated rational thinking" seems rare.

Anyway, I'm interested in hearing people's "rationalist backstories". Personally, I'm interested in reading really long and detailed backstories, but am also interested in reading "just a few paragraphs". I'm also eager to hear people's thoughts on my "extreme input/output" theory.

Rationality Reading Group: Part J: Death Spirals

4 Gram_Stone 24 September 2015 02:31AM

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Part J: Death Spirals (pp. 409-494)This post summarizes each article of the sequence, linking to the original LessWrong post where available.

J. Death Spirals

100. The Affect Heuristic - Positive and negative emotional impressions exert a greater effect on many decisions than does rational analysis.

101. Evaluability (and Cheap Holiday Shopping) - It's difficult for humans to evaluate an option except in comparison to other options. Poor decisions result when a poor category for comparison is used. Includes an application for cheap gift-shopping.

102. Unbounded Scales, Huge Jury Awards, and Futurism - Without a metric for comparison, estimates of, e.g., what sorts of punitive damages should be awarded, or when some future advance will happen, vary widely simply due to the lack of a scale.

103. The Halo Effect - Positive qualities seem to correlate with each other, whether or not they actually do.

104. Superhero Bias - It is better to risk your life to save 200 people than to save 3. But someone who risks their life to save 3 people is revealing a more altruistic nature than someone risking their life to save 200. And yet comic books are written about heroes who save 200 innocent schoolchildren, and not police officers saving three prostitutes.

105. Mere Messiahs - John Perry, an extropian and a transhumanist, died when the north tower of the World Trade Center fell. He knew he was risking his existence to save other people, and he had hope that he might be able to avoid death, but he still helped them. This takes far more courage than someone who dies, expecting to be rewarded in an afterlife for their virtue.

106. Affective Death Spirals - Human beings can fall into a feedback loop around something that they hold dear. Every situation they consider, they use their great idea to explain. Because their great idea explained this situation, it now gains weight. Therefore, they should use it to explain more situations. This loop can continue, until they believe Belgium controls the US banking system, or that they can use an invisible blue spirit force to locate parking spots.

107. Resist the Happy Death Spiral - You can avoid a Happy Death Spiral by (1) splitting the Great Idea into parts (2) treating every additional detail as burdensome (3) thinking about the specifics of the causal chain instead of the good or bad feelings (4) not rehearsing evidence (5) not adding happiness from claims that "you can't prove are wrong"; but not by (6) refusing to admire anything too much (7) conducting a biased search for negative points until you feel unhappy again (8) forcibly shoving an idea into a safe box.

108. Uncritical Supercriticality - One of the most dangerous mistakes that a human being with human psychology can make, is to begin thinking that any argument against their favorite idea must be wrong, because it is against their favorite idea. Alternatively, they could think that any argument that supports their favorite idea must be right. This failure of reasoning has led to massive amounts of suffering and death in world history.

109. Evaporative Cooling of Group Beliefs - When a cult encounters a blow to their own beliefs (a prediction fails to come true, their leader is caught in a scandal, etc) the cult will often become more fanatical. In the immediate aftermath, the cult members that leave will be the ones who were previously the voice of opposition, skepticism, and moderation. Without those members, the cult will slide further in the direction of fanaticism.

110. When None Dare Urge Restraint - The dark mirror to the happy death spiral is the spiral of hate. When everyone looks good for attacking someone, and anyone who disagrees with any attack must be a sympathizer to the enemy, the results are usually awful. It is too dangerous for there to be anyone in the world that we would prefer to say negative things about, over saying accurate things about.

111. The Robbers Cave Experiment The Robbers Cave Experiment, by Sherif, Harvey, White, Hood, and Sherif (1954/1961), was designed to investigate the causes and remedies of problems between groups. Twenty-two middle school aged boys were divided into two groups and placed in a summer camp. From the first time the groups learned of each other's existence, a brutal rivalry was started. The only way the counselors managed to bring the groups together was by giving the two groups a common enemy. Any resemblance to modern politics is just your imagination.

112. Every Cause Wants to Be a Cult - The genetic fallacy seems like a strange kind of fallacy. The problem is that the original justification for a belief does not always equal the sum of all the evidence that we currently have available. But, on the other hand, it is very easy for people to still believe untruths from a source that they have since rejected.

113. Guardians of the Truth - There is an enormous psychological difference between believing that you absolutely, certainly, have the truth, versus trying to discover the truth. If you believe that you have the truth, and that it must be protected from heretics, torture and murder follow. Alternatively, if you believe that you are close to the truth, but perhaps not there yet, someone who disagrees with you is simply wrong, not a mortal enemy.

114. Guardians of the Gene Pool - It is a common misconception that the Nazis wanted their eugenics program to create a new breed of supermen. In fact, they wanted to breed back to the archetypal Nordic man. They located their ideals in the past, which is a counterintuitive idea for many of us.

115. Guardians of Ayn Rand - Ayn Rand, the leader of the Objectivists, praised reason and rationality. The group she created became a cult. Praising rationality does not provide immunity to the human trend towards cultishness.

116. Two Cult Koans - Two Koans about individuals concerned that they may have joined a cult.

117. Asch's Conformity Experiment - The unanimous agreement of surrounding others can make subjects disbelieve (or at least, fail to report) what's right before their eyes. The addition of just one dissenter is enough to dramatically reduce the rates of improper conformity.

118. On Expressing Your Concerns - A way of breaking the conformity effect in some cases.

119. Lonely Dissent - Joining a revolution does take courage, but it is something that humans can reliably do. It is comparatively more difficult to risk death. But it is more difficult than either of these to be the first person in a rebellion. To be the only one who is saying something different. That doesn't feel like going to school in black. It feels like going to school in a clown suit.

120. Cultish Countercultishness - People often nervously ask, "This isn't a cult, is it?" when encountering a group that thinks something weird. There are many reasons why this question doesn't make sense. For one thing, if you really were a member of a cult, you would not say so. Instead, what you should do when considering whether or not to join a group, is consider the details of the group itself. Is their reasoning sound? Do they do awful things to their members?


This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

The next reading will cover Part K: Letting Go (pp. 497-532). The discussion will go live on Wednesday, 7 October 2015, right here on the discussion forum of LessWrong.

Against Expected Utility

-4 Houshalter 23 September 2015 09:21PM

Expected utility is optimal as the number of bets you take approaches infinity. You will lose bets on some days, and win bets on other days. But as you take more and more bets, the day to day randomness cancels out.

Say you want to save as many lives as possible. You can plug "number of lives saved" into an expected utility maximizer. And as the amount of bets it takes increases, it will start to save more lives than any other method.

But the real world obviously doesn't have an infinite number of bets. And following this algorithm in practice will get you worse results. It is not optimal.

In fact, as Pascal's Mugging shows, this could get arbitrarily terrible. An agent following expected utility would just continuously make bets with muggers and worship various religions, until it runs out of resources. Or worse, the expected utility calculations don't even converge, and the agent doesn't make any decisions.

So how do we fix it? Well we could just go back to the original line of reasoning that led us to expected utility, and fix it for finite cases. Instead of caring what method does the best on infinite bets, we might say we want the one that does the best the most on finite cases. That would get you median utility.

For most things, median utility will approximate expected utility. But for very very small risks, it will ignore them. It only cares that it does the best in most possible worlds. It won't ever trade away utility from the majority of your possible worlds to very very unlikely ones.

A naive implementation of median utility isn't actually viable, because at different points in time, the agent might make inconsistent decisions. To fix this, it needs to decide on policies instead of individual decisions. It will pick a decision policy which it believes will lead to the highest median outcome.

This does complicate making a real implementation of this procedure. But that's what you get when you generalize results, and try to make things work on the messy real world. Instead of idealized infinite worlds. The same issue occurs in the multi-armed bandit problem. Where the optimal infinite solution is simple, but finite solutions are incredibly complicated (or simple but require brute force.)

But if you do this, you don't need the independence axiom. You can be consistent and avoid money pumping without it. By not making decisions in isolation, but considering the entire probability space of decisions you will ever make. And choosing the best policies to navigate them.

It's interesting to note this actually solves some other problems. Such an agent would pick a policy that one-boxes on Newcomb's problems, simply because that is the optimal policy. Whereas a straightforward implementation of expected utility doesn't care.

But what if you really like the other mathematical properties of expected utility? What if we can just keep it and change something else? Like the probability function or the utility function?

Well the probability function is sacred IMO. Events should have the same probability of happening (given your prior knowledge), regardless what utility function you have, or what you are trying to optimize. And it's probably inconsistent too. An agent could exploit you. By giving you bets in the areas where your beliefs are forced to be different from reality.

The utility function is not necessarily sacred though. It is inherently subjective, with the goal of just producing the behavior we want. Maybe there is some modification to it that could fix these problems.

It seems really inelegant to do this. We had a nice beautiful system where you could just count the number of lives saved, and maximize that. But assume we give up on that. How can we change the utility function to make it work?

Well you could bound utility to get out of mugging situations. After a certain level, your utility function just stops. It can't get any higher.

But then you are stuck with a bound. If you ever reach it, then you suddenly stop caring about saving any more lives. Now it's possible that your true utility function really is bounded. But it's not a fully general solution for all utility functions. And I don't believe that human utility is actually bounded, but that will have to be a different post.

You could transform the utility function so it asymptotic. But this is just a continuous bound. It doesn't solve much. It still makes you care less and less about obtaining more utility, the closer you get to it.

Say you set your asymptote around 1,000. It can be much larger, but I need an example that is manageable. Now, what happens if you find yourself to exist in a world where all utilities are multiplied by a large number? Say 1,000. E.g. you save a 1,000 lives in situations where before, you would have saved only 1.

An example asymptoting function that is capped at 1,000. Notice how 2,000 is only slightly higher than 1,000, and everything after that is basically flat.

Now the utility of each additional life is diminishing very quickly. Saving 2,000 lives might have only 0.001% more utility than 1,000 lives.

This means that you would not take a 1% risk of losing 1,000 people, for a 99% chance at saving 2,000.

This is the exact opposite situation of Pascal's mugging! The probability of the reward is very high. Why are we refusing such an obviously good trade?

What we wanted to do was make it ignore really low probability bets. What we actually did was just make it stop caring about big rewards, regardless of the probability.

No modification to it can fix that. Because the utility function is totally indifferent to probability. That's what the decision procedure is for. That's where the real problem is.

In researching this topic I've seen all kinds of crazy resolutions to Pascal's Mugging. Some try to attack the exact thought experiment of an actual mugger. And miss the general problem of low probability events with large rewards. Others try to come up with clever arguments why you shouldn't pay the mugger. But not any general solution to the problem. And not one that works under the stated premises, where you care about saving human lives equally, and where you assign the mugger less than 1/3↑↑↑3 probability.

In fact Pascal's Mugger was originally written just to be a formalization of Pascal's original wager. Pascal's wager was dismissed for reasons like involving infinite utilities, and the possibility of an "anti-god" that exactly cancels the benefits out. Or that God wouldn't reward fake worshippers. People mostly missed the whole point about whether or not you should take low probability, high reward bets.

Pascal's Mugger showed that, no, it works fine in finite cases, and the probabilities do not have to exactly cancel each other out

Some people tried to fix the problem by adding hacks on top of the probability or utility functions. I argued against these solutions above. The problem is fundamentally with the decision procedure of expected utility.

I've spoken to someone who decided to just bite the bullet. He accepted that our intuition about big numbers is probably wrong, and we should just do what the math tells us.

But even that doesn't work. One of the points made in the original Pascal's Mugging post is that EU doesn't even converge. There is a hypothesis which has even less probability than the mugger, but promises 3↑↑↑↑3 utility. And a hypothesis even smaller than that which promises 3↑↑↑↑↑3 utility, and so on. Expected utility is utterly dominated by increasingly more improbable hypotheses. The expected utility of all actions approaches positive or negative infinity.

Expected utility is at the heart of the problem. We don't really want the average of our utility function over all possible worlds. No matter how big the numbers are or improbable they may be. We don't really want to trade away utility from the majority of our probability mass to infinitesimal slices of it.

The whole justification for EU being optimal in the infinite case, doesn't apply to the finite real world. The axioms that imply you need it to be consistent aren't true if you don't assume independence. So it's not sacred, and we can look at alternatives.

Median utility is just a first attempt at an alternative. We probably don't really want to maximize median utility either. Stuart Armstrong suggests using the mean of quantiles. There are probably better methods too. In fact there is an entire field of summary statistics and robust statistics, that I've barely looked at yet.

We can generalize and think of agents has having two utility functions. The regular utility function, which just gives a numerical value representing how preferable an outcome is. And a probability preference function, which gives a numerical value to each probability distribution of utilities.

Imagine we want to create an AI which acts the same as the agent would, given the same knowledge. Then we would need to know both of these functions. Not just the utility function. And they are both subjective, with no universally correct answer. Any function, so long as it converges (unlike expected utility), should produce perfectly consistent behavior.

View more: Next