One of the most powerful rationalist techniques is noticing your surprise. 

It ties in to several deep issues. One of them relates to one of my favorite LW comments  (the second highest upvoted one in Main):

One of the things that I've noticed about this is that most people do not expect to understand things. For most people, the universe is a mysterious place filled with random events beyond their ability to comprehend or control. Think "guessing the teacher's password", but not just in school or knowledge, but about everything.

Such people have no problem with the idea of magic, because everything is magic to them, even science.

--pjeby

For the universe to make sense to you, you have to have a model; for that model to be useful, you have to notice what it says, and then you need to act on it. I've done many things the wrong way in my life, but the ones I remember as mistakes are the ones where some part of me *knew* it was a problem, and instead of having a discussion with that part of me, I just ignored it and marched on.

It is good to notice your surprise. But that's only the first step.

--Douglas_Knight

 

So any stories, of tricks you noticed, didn't notice, or successfully pulled?

Self-motivated hard work is the primary source of the intense, optimistic engagement known as flow—one of the greatest forms of happiness that makes us come alive with purpose and potential (Csikszentmihalyi, 1975). Sadly, for most people work does not feel so rewarding most of the time. Instead we often have to persevere through long periods of hard, painful, and unrewarding work when we could be doing something much more enjoyable. When faced with this motivational challenge people often give up too easily, get sidetracked, or procrastinate (Steel, 2007). The problem is not that we are not willing or unable to work hard. To the contrary, we crave being productively engaged in challenging tasks. Thus, instead of blaming ourselves for our limited will-power, it may be more productive to take a critical look at the carrots and the sticks that are supposed to help us stay motivated. Who put them there and why? Are these incentives helpful, distracting, irrelevant, or out of sight? If you could place them differently and add new ones, where would they go? Often, the problem is that the rewards we experience in the short run are misaligned with what we want to accomplish. In the short run the extremely valuable work that brings us closer to our cherished goals can be aversive while activities that are irrelevant or even opposed to everything we want to accomplish can be pleasant and rewarding. Hence, when we struggle to be engaged with something that we care about, then perhaps we are not the problem but the incentives are, or as Jane McGonigal (2011) put it "Reality is broken".

So, if reality is broken, then what can we do to fix it?  One approach is to design better incentive structures that make the pursuit of our goals more engaging. If we want to go this way, then there is a lot to be learned from games, because their incentive structures are so well designed that they let people enjoy hard work for many hours on end (McGonigal, 2011). In the past five years, the success of video games has inspired the gamification of education, work, health, and business. Gamification is the use of game elements, like points, levels, badges, and quests to engage, motivate, and nudge people in non-game contexts. There are even tools like SuperBetter and Habitica that individuals like you and I can use to gamify our own lives. Previous studies have shown that gamification can have positive effects on motivation, engagement, behavior, learning outcomes, and health—but only when it is done right (Hamari, Koivisto, & Sarsa, 2014; Roepke, et al., 2015). But when gamification is done wrong it can have negative effects by incentivizing counter-productive behaviors. So far gamification has been an art, and there is very little science about how to do it right. This motivated my advisor and me to develop a practical theory of optimal gamification.

In this blog post I focus on how our theory could be applied in practice. If you would like to learn about the technical details or read more about our experiments, then please take look at our CogSci paper (Lieder & Griffiths, submitted). I will start with a very brief summary of our method, provide an intuitive explanation of what it does, and then dive into how you can implement it in your own life. I will close with an outlook on how our method could be applied to gamify our todo lists.

Level 1: Optimal Gamification

Our method for optimal gamification draws on the theory of Markov decision processes (MDPs; Sutton & Barto, 1998) and the shaping theorem (Ng, Harada, & Russell, 1999). The basic idea is to align each action's immediate reward with its value in the long run. Therefore the points should complement the immediate rewards of doing something (e.g., how painful it is) by the value that it generates in the long run. Concretely, the points awarded for an activity should be chosen such that the right thing to do looks best in the short run when you combine how many points it is worth with how it feels when you do it. Furthermore, the points have to be assigned in such a way that when you undo something you lose as many points as you earned when you did it. We evaluated the effectiveness of our method in two behavioral experiments. Our first experiment demonstrated that incentive structures designed by our method can indeed help people make better, less short-sighted decisions—especially when course of action that is best in the long run is unpleasant in the short run. We also found that less principled approaches to gamification can encourage ruthless rushing towards a goal that causes more harm than good, and we showed that our method is guaranteed to avoid these perils. In the second experiment we found that the optimal incentive structures designed with our method can be effectively implemented using game elements like points and badges. These results suggest that the proposed method provides a principled way to leverage gamification to help people make better decisions.

Our method proceeds in three steps:

1.    Model the situation and the decision-maker's goals and options as a MDP.

2.    Solve the MDP to obtain the optimal value function V* or approximate it.

3.    Set the number of points for progressing from stage s to stage s' to V*(s')-V*(s).  

Intuitively, this means that the number of points that is awarded for doing something should reflect how much better the resulting state (i.e., s') is than the previous one (i.e., s). For instance, achieving a goal is worth 1000 points then completing 10% of the work required to reach the goal should be rewarded with 100 points. So let's think about how you could apply this approach right now without having to solve MDPs.

Level 2: Practical Implications

In my day-to-day life I try to approximate optimal gamification as follows:

1.    Set a concrete goal that you would like to achieve and figure out how many points it is worth, e.g. writing this blog post was worth 1000 points to me.

2.    Set several milestones along the way to the goal to divide the path into small steps that feel very manageable.

3.    For each milestone, determine how far you will have come when you get there as a percentage of the total distance to the goal, e.g. 10%, 20%, 30%, ..., 100% for the first, second, third, ..., and the tenth milestone respectively.

4.    Assign each milestone the corresponding fraction of the total value of achieving the goal, e.g. 100 points, 200 points, 300 points, ..., and 1000 points for the first, second, third, ..., and tenth milestone respectively.

5.    Figure out what you have to do to get from one milestone to the next. If this is a simple activity, then its reward should be the difference between the value of next milestone and the value of the current milestone, e.g. 100 points. If it is a complex sequence of actions, then make it a subgoal and apply steps 1-3 figure out how to achieve it.

6.    Once you are done with step 5, you can add those points to your todo-list.

7.    Now it is time to get things done and reward yourself. You start at 0 points, but whenever you complete one of the steps, you earn as many points as you have assigned to it and can increment your (daily) score.

Earning these points can be very rewarding if you remind yourself what they stand for. If your goal was worth $1,000,000 to you and you assigned 1000 points to it, then 10 points should be worth $10,000 to you. But if this is not rewarding enough for you, you can think of ways that make the points more pleasurable. You could, for instance, make a high-score list that motivates you to beat your personal best day after day or start a high-score competition with your friends. You could also set yourself the goal to achieve a certain number of points by a certain time and promise yourself a treat if you achieve it.

There are many other ways that you could assign points to the items on you todo list. Feel free to do whatever works for you. But it may be useful to keep in mind that the way in which optimal gamification assigns points has several formal properties that are necessary to avoid negative side-effects:

a) Each item's score reflects how valuable is in the long run.

Optimal gamification works because it aligns each action's immediate reward with its long-term value. To help you make better decisions the points should be designed such that the course of action that is best in the long run looks best in the short run. This entails incentivizing unpleasant or unrewarding activities that will pay off later—especially when their less productive alternatives are very rewarding in the short run.

b) Beware of cycles!

The shaping theorem (Ng, et al., 1999) requires that going back and forth between two states receives a net pseudo-reward of zero. When your pseudo-rewards along a circle add up to a positive value, then you may be incentivizing yourself to create unnecessary problems for yourself. This can happen when the action for which you reward yourself can only be executed in an undesirable state, and you do not equally punish yourself for falling back into that state. For instance adding points for losing weight will inadvertently incentivize you to regain weight afterwards unless you subtract at least the same number of points for gaining weight. Similarly, if you reward yourself for solving interpersonal conflicts but don’t punish yourself for creating them, then you may be setting yourself up for trouble. To avoid such problems, creating a problem must be punished by at least as many points as you earn by solving it. 

c) Two ways to achieve the same goal should yield the same number of points.

The shaping theorem also requires that all paths that lead to the same final state (e.g., having submitted a paper by the deadline) should yield the same amount of reward. If this is not the case your pseudo-rewards may bias you towards a suboptimal path. For instance, if you reward your all-nighter on the last night before the deadline by the reward value of a month’s worth of work, you are incentivizing yourself to procrastinate. Similarly, if you reward one activity that leads towards your goal much more heavily than others, then you may be biasing yourself towards a reckless course of action that may achieve the goal at an unreasonably high cost. For instance, rewarding yourself 100 times as much for working 100% on a project than for working on it 50% might lead you to complete the project early at the expense of your health, your friendships, your education, and all your other projects. To avoid this problem, al paths that lead to the same state should yield the same amount of reward. 

d) Pseudo-rewards should be awarded for state-transitions instead of actions.

Many applications of gamification reward "good" actions with points regardless of when or how often these actions are taken. But according to the shaping theorem, the number of points must depend on the state in which the action is taken and the state that it leads to. If your pseudo-rewards were based only on what you do but not on when you do it, then you might keep rewarding yourself for something even when it is no longer valuable, because the underlying state has changed. For instance, at some point your reward for losing weight has to diminish or else you may be setting yourself up for anorexia.

Level 3: Todo-list gamification

My first practical application is to manually gamify my todo-list every morning. I find this very helpful and motivating: Assigning points to the items on my todo list makes me realize how much I value them. This is useful for prioritizing important task. Earning points allows me to perceive my progress more more accurately and more vividly. This helps me feel great about getting something important done even when it was only a single item on my todo list and took me a lot of time and effort to accomplish. Conversely, the point scheme also prevents me from feeling so good about checking off small things that I become tempted to neglect the big ones that are much more important. Gamification thereby remedies the todo list's shortcoming that it makes each item seem equally important. I highly recommend gamifying your todo lists. It can be highly motivating. Yet, adding the points manually takes some effort and my point scheme is often somewhat arbitrary and probably suboptimal.

To make todo list gamification easier and more effective, I am planning to develop an easy-to-use website or app that will do optimal gamification for you. Its graphical user interface would allow you to create hierarchical todo-lists, ask you 1 or 2 simple questions about each item on your list and then gamify your todo-list for you. To do this, it will translate your list and your answers into a MDP, compute its optimal value function, and use it to determine how valuable it is to complete each item. The tool could also help you set manageable subgoals and determine what is most important and should be done first. Last but not least, a website or app can also leverage additional game elements to make the points that you earn more rewarding: It can track your productivity and provide instant feedback that makes your progress more salient. It can send you on a quest that gives you a goals along with small actionable steps. The tool could allow you to realize that you are getting ever more productive by visualizing your progress over time. As you become more effective, you level up and your quests will become increasingly more challenging.  It might include a scoreboard that lets you compete with yourself and/or others and win prizes for your performance. Last but not least, if you need an extra push, you can tie your points to social rewards, your favorite treat, money, or access to your favorite music, apps, or websites. There are many more possibilities, and I invite you to think about it and share your ideas. In brief, there is wealth of opportunities to leverage game elements to make goal achievement fun and easy.

Join me on my quest! An adventure awaits. 

Gamification can be a useful tool to make achieving your goals easier and more engaging. However, gamification only works when it is done right. The theory of MDPs and pseudo-rewards provide the formal tools needed to do gamification right. With the help of these tools we can design incentive structures that help people overcome motivational obstacles, do the right thing and achieve their goals. But more research and development needs to be done to make optimal gamification practical.

If you have any thoughts or ideas for what to do next, noticed a problem with the approach, or would like to be part of our team and contribute to building a tool helps people achieve their goals, please send me an e-mail.

References and recommended readings

Csikszentmihalyi, M. (1975). Beyond boredom and anxiety: the experience of play in work and games. San Francisco: Jossey-Bass.

Lieder, F., & Griffiths, T.L. (submitted). Helping people make better decisions using optimal gamification. CogSci 2016. [Manuscript]

McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. New York: Penguin.

McGonigal, J. (2015). SuperBetter: A revolutionary approach to getting stronger, happier, braver and more resilient–powered by the science of games. London, UK: Penguin Press.

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work?–A literature review of empirical studies on gamification. In 47th Hawaii international conference on system sciences (pp. 3025–3034).

Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In I. Bratko & S. Dzeroski (Eds.), Proceedings of the 16th annual international conference on machine learning (Vol. 16, pp. 278–287). San Francisco, CA, USA: Morgan Kaufmann.

Roepke, A. M., Jaffee, S. R., Riffle, O. M., McGonigal, J., Broome, R., & Maxwell, B. (2015). Randomized controlled trial of SuperBetter, a smartphone-based/Internet-based self-help tool to reduce depressive symptoms. Games for health journal, 4(3), 235-246.

Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA, USA: MIT press.

 

I like to read posts on "Main" from time to time, including ones that haven't been promoted. However, lately, these posts get drowned out by all the meetup announcements.

It seems like this could lead to a cycle where people comment less on recent non-promoted posts (because they fall off the Main non-promoted area quickly) which leads to less engagement, and less posts, etc.

Meetups are also very important, but here's the rub: I don't think a text-based announcement in the Main area is the best possible way to showcase meetups.

So here's an idea: how about creating either a calendar of upcoming meetups, or map with pins on it of all places having a meetup in the next three months?

This could be embedded on the front page of leswrong.com -- that'd let people find meetups easier (they can look either by timeframe or see if their region is represented), and would give more space to new non-promoted posts, which would hopefully promote more discussion, engagement, and new posts.

Thoughts?

DeepMind's go AI, called AlphaGo, has beaten the European champion with a score of 5-0. A match against top ranked human, Lee Se-dol, is scheduled for March.

 

Games are a great testing ground for developing smarter, more flexible algorithms that have the ability to tackle problems in ways similar to humans. Creating programs that are able to play games better than the best humans has a long history

[...]

But one game has thwarted A.I. research thus far: the ancient game of Go.

[Cross]

Imagine this:

Oliver: … Thus we see that donating to the opera is the best way of promoting the arts.

Eleanor: Okay, but I’m principally interested in improving human welfare.

Oliver: Oh! Well I think it is also the case that donating to the opera is best for improving human welfare too.

Generally, what is best for one thing is usually not the best for something else, and thus Oliver’s claim that donations to opera are best for the arts and human welfare is surprising. We may suspect bias: that Oliver’s claim that the Opera is best for the human welfare is primarily motivated by his enthusiasm for opera and desire to find reasons in favour, rather than a cooler, more objective search for what is really best for human welfare.

The rest of this essay tries to better establish what is going on (and going wrong) in cases like this. It is in three parts: the first looks at the ‘statistics’ of convergence - in what circumstances is it surprising to find one object judged best by the lights of two different considerations? The second looks more carefully at the claim of bias: how it might be substantiated, and how it should be taken into consideration. The third returns to the example given above, and discusses the prevalence of this sort of error ‘within’ EA, and what can be done to avoid it.

Varieties of convergence

Imagine two considerations, X and Y, and a field of objects to be considered. For each object, we can score it by how well it performs by the lights of the considerations of X and Y. We can then plot each object on a scatterplot, with each axis assigned to a particular consideration. How could this look?

At one extreme, the two considerations are unrelated, and thus the scatterplot shows no association. Knowing how well an object fares by the lights of one consideration tells you nothing about how it fares by the lights of another, and the chance that the object that scores highest on consideration X also scores highest on consideration Y is very low. Call this no convergence.

At the other extreme, considerations are perfectly correlated, and the ‘scatter’ plot has no scatter, but rather a straight line. Knowing how well an object fares by consideration X tells you exactly how well it fares by consideration Y, and the object that scores highest on consideration X is certain to be scored highest on consideration Y. Call this strong convergence.

In most cases, the relationship between two considerations will lie between these extremes: call this weak convergence. One example is there being a general sense of physical fitness, thus how fast one can run and how far one can throw are somewhat correlated. Another would be intelligence: different mental abilities (pitch discrimination, working memory, vocabulary, etc. etc.) all correlate somewhat with one another.

More relevant to effective altruism, there also appears to be weak convergence between different moral theories and different cause areas. What is judged highly by (say) Kantianism tends to be judged highly by Utilitarianism: although there are well-discussed exceptions to this rule, both generally agree that (among many examples) assault, stealing, and lying are bad, whilst kindness, charity, and integrity are good.(1) In similarly broad strokes what is good for (say) global poverty is generally good for the far future, and the same applies for between any two ‘EA’ cause areas.(2)

In cases of weak convergence, points will form some some sort of elliptical scatter, and knowing how an object scores on X does tell you something about how well it scores on Y. If you know that something scores highest for X, your expectation of how it scores for Y should go upwards, and the chance of it also scores highest for Y should increase. However, the absolute likelihood of it being best for X and best for Y remains low, for two main reasons:

Trade-offs: Although consideration X and Y are generally positively correlated, there might be a negative correlation at the far tail, due to attempts to optimize for X or Y  at disproportionate expense for Y or X. Although in the general population running and throwing will be positively correlated with one another, elite athletes may optimize their training for one or the other, and thus those who specialize in throwing and those who specialize in running diverge. In a similar way, we may think believe there is scope for similar optimization when it comes to charities or cause selection.

Chance: (c.f.) Even in cases where there are no trade-offs, as long as the two considerations are somewhat independent, random fluctuations will usually ensure the best by consideration X will not be best by consideration Y. That X and Y only weakly converge implies other factors matter for Y besides X. For the single object that is best for X, there will be many more not best for X (but still very good), and out of this large number of objects it is likely one will do very well on these other factors to end up the best for Y overall. Inspection of most pairs of correlated variables confirms this: Those with higher IQ scores tend to be wealthier, but the very smartest aren’t the very wealthiest (and vice versa), serving fast is good for tennis, but the very fastest servers are not the best players (and vice versa), and so on. Graphically speaking, most scatter plots bulge in an ellipse rather than sharpen to a point.

The following features make a single object scoring highest on two considerations more likely:

1. The smaller the population of objects. Were the only two options available to OIiver and Eleanor, “Give to the Opera” and “Punch people in the face”, it is unsurprising the former comes top for many considerations.
2. The strength of their convergence. The closer the correlation moves to collinearity, the less surprising finding out something is best for both. It is less surprising the best at running 100m is best at running 200m, but much more surprising if it transpired they threw discus best too.
3. The ‘wideness’ of the distribution. The heavier the tails, the more likely a distribution is to be stretched out and ‘sharpen’ to a point, and the less likely bulges either side of the regression line are to be populated. (I owe this to Owen Cotton-Barratt)

In the majority of cases (including those relevant to EA), there is a large population of objects, weak convergence and (pace the often heavy-tailed distributions implicated) it is uncommon for one thing to be best b the lights of two weakly converging considerations.

Proxy measures and prediction

In the case that we have nothing to go on to judge what is good for Y save knowing what is good for X. Our best guess for what is best for Y is what is best for X. Thus the Opera is the best estimate for what is good for human welfare, given only the information that it is best for the arts. In this case, we should expect our best guess to be very likely wrong. Although it is more likely than any similarly narrow alternative (“donations to the opera, or donations to X-factor?”) Its absolute likelihood relative to the rest of the hypothesis space is very low (“donations to the opera, or something else?”).

Of course, we usually have more information available. Why not search directly for what is good for human welfare, instead of looking at what is good for the arts? Often searching for Y directly rather than a weakly converging proxy indicator will do better: if one wants to select a relay team, selecting based on running speed rather than throwing distance looks a better strategy. Thus finding out a particular intervention (say the Against Malaria Foundation) comes top when looking for what is good for human welfare provides much stronger evidence it is best for human welfare than finding out the opera comes top when looking for what is good for a weakly converging consideration.(3)

Pragmatic defeat and Poor Propagation

Eleanor may suspect bias is driving Oliver’s claim on behalf of the opera. The likelihood of the opera being best for both the arts and human welfare is low, even taking their weak convergence into account. The likelihood of bias and motivated cognition colouring Oliver’s judgement is higher, especially if Oliver has antecedent commitments to the opera. Three questions: 1) Does this affect how she should regard Oliver’s arguments? 2) Should she keep talking to Oliver, and, if she does, should she suggest to him he is biased? 3) Is there anything she can do to help ensure she doesn’t make a similar mistake?

Grant Eleanor is right that Oliver is biased. So what? It entails neither he is wrong nor the arguments he offers in support are unsound: he could be biased and right. It would be a case of the genetic fallacy (or perhaps ad hominem) to argue otherwise. Yet this isn’t the whole story: informal ‘fallacies’ are commonly valuable epistemic tools; we should not only attend to the content of arguments offered, but argumentative ‘meta-data’ such as qualities of the arguer as well.(4)

Consider this example. Suppose you are uncertain whether God exists. A friendly local Christian apologist offers the reasons why (in her view) the balance of reason clearly favours Theism over Atheism. You would be unwise to judge the arguments purely ‘on the merits’: for a variety of reasons, the Christian apologist is likely to have slanted the evidence she presents to favour Theism; the impression she will give of where the balance of reason lies will poorly track where the balance of reason actually lies. Even if you find her arguments persuasive, you should at least partly discount this by what you know of the speaker.

In some cases it may be reasonable to dismiss sources ‘out of hand’ due to their bias without engaging on the merits: we may expect the probative value of the reasons they offer, when greatly attenuated by the anticipated bias, to not be worth the risks of systematic error if we mistake the degree of bias (which is, of course, very hard to calculate); alternatively, it might just be a better triage of our limited epistemic resources to ignore partisans and try and find impartial sources to provide us a better view of the balance of reason.

So: should Eleanor stop talking to Oliver about this topic? Often, no. First (or maybe zeroth), there is the chance she is mistaken about Oliver being biased, and further discussion would allow her to find this out. Second, there may be tactical reasons: she may want to persuade third parties to their conversation. Third, she may guess further discussion is the best chance of persuading Oliver, despite the bias he labours under. Fourth, it may still benefit Eleanor: although bias may undermine the strength of reasons Oliver offers, they may still provide her with valuable information. Being too eager to wholly discount what people say based on assessments of bias (which are usually partly informed by object level determinations of various issues) risks entrenching one’s own beliefs.

Another related question is whether it is wise for Eleanor to accuse Oliver of bias. There are some difficulties. Things that may bias are plentiful, thus counter-accusations are easy to make: (“I think you’re biased in favour of the opera due to your prior involvement”/”Well, I think you’re biased against the opera due to your reductionistic and insufficiently holistic conception of the good.”) They are apt to devolve into the personally unpleasant (“You only care about climate change because you are sleeping with an ecologist”) or the passive-aggressive (“I’m getting really concerned that people who disagree with me are offering really bad arguments as a smokescreen for their obvious prejudices”). They can also prove difficult to make headway on. Oliver may assert his commitment was after his good-faith determination that opera really was best for human welfare and the arts. Many, perhaps most, claims like these are mistaken, but it can be hard to tell (or prove) which.(5)

Eleanor may want to keep an ‘internal look out’ to prevent her making a similar mistake to Oliver. One clue is a surprising lack of belief propagation: we change our mind about certain matters, and yet our beliefs about closely related matters remain surprisingly unaltered. In most cases where someone becomes newly convinced of (for example) effective altruism, we predict this should propagate forward and effect profound changes to their judgements on where to best give money or what is the best career for them to pursue. If Eleanor finds in her case that this does not happen, that in her case her becoming newly persuaded by the importance of the far future does not propagate forward to change her career or giving, manifesting instead in a proliferation of ancillary reasons that support her prior behaviour, she should be suspicious of this surprising convergence between what she thought was best then, and what is best now under considerably different lights.

EA examples

Few Effective altruists seriously defend the opera as a leading EA cause. Yet the general problem of endorsing surprising and suspicious convergence remains prevalent. Here are some provocative examples:

1. The lack of path changes. Pace personal fit, friction, sunk capital, etc. it seems people who select careers on ‘non EA grounds’ often retain them after ‘becoming’ EA, and then provide reasons why (at least for them) persisting in their career is the best option.
2. The claim that, even granting the overwhelming importance of the far future, it turns out that animal welfare charities are still the best to give to, given their robust benefits, positive flow through effects, and the speculativeness of far future causes.
3. The claim that, even granting the overwhelming importance of the far future, it turns out that global poverty charities are still the best to give to, given their robust benefits, positive flow through effects, and the speculativeness of far future causes.
4. Claims from enthusiasts of Cryonics or anti-aging research that this, additional to being good for their desires for an increased lifespan, is also a leading ‘EA’ buy.
5. A claim on behalf of veganism that it is the best diet for animal welfare and for the environment and for individual health and for taste.

All share similar features: one has prior commitments to a particular cause area or action. One becomes aware of a new consideration which has considerable bearing on these priors. Yet these priors don’t change, and instead ancillary arguments emerge to fight a rearguard action on behalf of these prior commitments - that instead of adjusting these commitments in light of the new consideration, one aims to co-opt the consideration to the service of these prior commitments.

Naturally, that some rationalize doesn’t preclude others being reasonable, and the presence of suspicious patterns of belief doesn’t make them unwarranted. One may (for example) work in global poverty due to denying the case for the far future (via a person affecting view, among many other possibilities) or aver there are even stronger considerations in favour (perhaps an emphasis on moral uncertainty and peer disagreement and therefore counting the much stronger moral consensus around stopping tropical disease over (e.g.) doing research into AI risk as the decisive consideration).

Also, for weaker claims, convergence is much less surprising. Were one to say on behalf of veganism: “It is best for animal welfare, but also generally better for the environment and personal health than carnivorous diets. Granted, it does worse on taste, but it is clearly superior all things considered”, this seems much less suspect (and also much more true) than the claim it is best by all of these metrics. It would be surprising if the optimal diet for personal health did not include at least some animal products.

Caveats aside, though, these lines of argument are suspect, and further inspection deepens these suspicions. In sketch, one first points to some benefits the prior commitment has by the lights of the new consideration (e.g. promoting animal welfare promotes antispeciesism, which is likely to make the far future trajectory go better), and second remarks about how speculative searching directly on the new consideration is (e.g. it is very hard to work out what we can do now which will benefit the far future).(6)

That the argument tends to end here is suggestive of motivated stopping. For although the object level benefits of (say) global poverty are not speculative, their putative flow-through benefits on the far future are speculative. Yet work to show that this is nonetheless less speculative than efforts to ‘directly’ work on the far future is left undone.(7) Similarly, even if it is the case the best way to make the far future go better is to push on a proxy indicator, which one? Work on why (e.g.) animal welfare is the strongest proxy out of competitors also tends to be left undone.(8) As a further black mark, it is suspect that those maintaining global poverty is the best proxy almost exclusively have prior commitments to global poverty causes, mutatis mutandis animal welfare, and so on.

We at least have some grasp of what features of (e.g.) animal welfare interventions make them good for the far future. If this (putatively) was the main value of animal welfare interventions due to the overwhelming importance of the far future, it would seem wise to try and pick interventions which maximize these features. So we come to a recursion: within animal welfare interventions, ‘object level’ and ‘far future’ benefits would be expected to only weakly converge. Yet (surprisingly and suspiciously) the animal welfare interventions recommended by the lights of the far future are usually the same as those recommended on ‘object level’ grounds.

Conclusion

If Oliver were biased, he would be far from alone. Most of us are (like it or not) at least somewhat partisan, and our convictions are in part motivated by extra-epistemic reasons: be it vested interests, maintaining certain relationships, group affiliations, etc. In pursuit of these ends we defend our beliefs against all considerations brought to bear against them. Few beliefs are indefatigable by the lights of any reasonable opinion, and few policy prescriptions are panaceas. Yet all of ours are.

It is unsurprising the same problems emerge within effective altruism: a particular case of ‘pretending to actually try’ is ‘pretending to take actually arguments seriously’.(9)These problems seem prevalent across the entirety of EA: that I couldn’t come up with good examples for meta or far future cause areas is probably explained by either bias on my part or a selection effect: were these things less esoteric, they would err more often.(10)

There’s no easy ‘in house’ solution, but I repeat my recommendations to Eleanor: as a rule, maintaining dialogue, presuming good faith, engaging on the merits, and listening to others seems a better strategy, even if we think bias is endemic. It is also worth emphasizing the broad (albeit weak) convergence between cause areas is fertile common ground, and a promising area for moral trade. Although it is unlikely that the best thing by the lights of one cause area is the best thing by the lights of another, it is pretty likely it will be pretty good. Thus most activities by EAs in a particular field should carry broad approbation and support from those working in others.

I come before you a sinner too. I made exactly the same sorts of suspicious arguments myself on behalf of global poverty. I’m also fairly confident my decision to stay in medicine doesn’t really track the merits either – but I may well end up a beneficiary of moral luck. I’m loath to accuse particular individuals of making the mistakes I identify here. But, insofar as readers think this may apply to them, I urge them to think again.(11)

Notes

1. We may wonder why this is the case: the content of the different moral theories are pretty alien to one another (compare universalizable imperatives, proper functioning, and pleasurable experiences). I suggest the mechanism is implicit selection by folk or ‘commonsense’ morality. Normative theories are evaluated at least in part by how well they accord to our common moral intuitions, and they lose plausibility commensurate to how much violence they do to them. Although cases where a particular normative theory apparently diverges from common sense morality are well discussed (consider Kantianism and the inquiring murder, or Utilitarianism and the backpacker), moral theories that routinely contravene our moral intuitions are non-starters, and thus those that survive to be seriously considered somewhat converge with common moral intuitions, and therefore one another.
2. There may be some asymmetry: on the object level we may anticipate the ‘flow forward’ effects of global health on x-risk to be greater than the ‘flow back’ benefits of x-risk work on global poverty. However (I owe this to Carl Shulman) the object level benefits are probably much smaller than more symmetrical ‘second order’ benefits, like shared infrastructure, communication and cross-pollination, shared expertise on common issues (e.g. tax and giving, career advice).
3. But not always. Some things are so hard to estimate directly, and using proxy measures can do better. The key question is whether the correlation between our outcome estimates and the true values is greater than that between outcome and (estimates of) proxy measure outcome. If so, one should use direct estimation; if not, then the proxy measure. There may also be opportunities to use both sources of information in a combined model.
4. One example I owe to Stefan Schubert: we generally take the fact someone says something as evidence it is true. Pointing out relevant ‘ad hominem’ facts (like bias) may defeat this presumption.
5. Population data – epistemic epidemiology, if you will – may help. If we find that people who were previously committed to the operas much more commonly end up claiming the opera is best for human welfare than than other groups, this is suggestive of bias.

   A subsequent problem is how to disentangle bias from expertise or privileged access. Oliver could suggest that those involved in the opera gain ‘insider knowledge’, and their epistemically superior position explains why they disproportionately claim the opera is best for human welfare.

   Some features can help distinguish between bias and privileged access, between insider knowledge and insider beliefs. We might be able to look at related areas, and see if ‘insiders’ have superior performance which an insider knowledge account may predict (if insiders correctly anticipate movements in consensus, this is suggestive they have an edge). Another possibility is to look at migration of beliefs. If there is ‘cognitive tropism’, where better cognizers tend to move from the opera to AMF, this is evidence against donating to the opera in general and the claim of privileged access among opera-supporters in particular. Another is to look at ordering: if the population of those ‘exposed’ to the opera first and then considerations around human welfare are more likely to make Oliver’s claims than those exposed in reverse order, this is suggestive of bias on one side or the other.

6. Although I restrict myself to ‘meta’-level concerns, I can’t help but suggest the ‘object level’ case for these things looks about as shaky as Oliver’s object level claims on behalf of the opera. In the same way we could question: “I grant that the arts is the an important aspect of human welfare, but is it the most important (compared to, say, avoiding preventable death and disability)?” or “What makes you so confident donations to the opera are the best for the arts - why not literature? or perhaps some less exoteric music?” We can post similarly tricky questions to proponents of 2-4: “I grant that (e.g.) antispeciesism is an important aspect of making the far future go well, but is it the most important aspect (compared to, say, extinction risks)?” or “What makes you so confident (e.g) cryonics is the best way of ensuring greater care for the future - what about militating for that directly? Or maybe philosophical research into whether this is the correct view in the first place?”

   It may well be that there are convincing answers to the object level questions, but I have struggled to find them. And, in honesty, I find the lack of public facing arguments in itself cause for suspicion.

7. At least, undone insofar as I have seen. I welcome correction in the comments.
8. The only work I could find taking this sort of approach is this.
9. There is a tension between ‘taking arguments seriously’ and ‘deferring to common sense’. Effective altruism only weakly converges with common sense morality, and thus we should expect their recommendations to diverge. On the other hand, that something lies far from common sense morality is a pro tanto reason to reject it. This is better acknowledged openly: “I think the best action by the lights of EA is to research wild animal suffering, but all things considered I will do something else, as how outlandish this is by common sense morality is a strong reason against it”. (There are, of course, also tactical reasons that may speak against saying or doing very strange things.)
10. This ‘esoteric selection effect’ may also undermine social epistemological arguments between cause areas:

    It seems to me that more people move from global poverty to far future causes than people move in the opposite direction (I suspect, but am less sure, the same applies between animal welfare and the far future). It also seems to me that (with many exceptions) far future EAs are generally better informed and cleverer than global poverty EAs.

    I don’t have great confidence in this assessment, but suppose I am right. This could be adduced as evidence in favour of far future causes: if the balance of reason favoured the far future over global poverty, this would explain the unbalanced migration and ‘cognitive tropism’ between the cause areas.

    But another plausible account explains this by selection. Global poverty causes are much more widely known that far future causes. Thus people who are ‘susceptible’ to be persuaded by far future causes were often previously persuaded by global poverty causes, whilst the reverse is not true - those susceptible to global poverty causes are unlikely to encounter far future causes first. Further, as far future causes are more esoteric, they will be disproportionately available to better-informed people. Thus, even if the balance of reason was against the far future, we would still see these trends and patterns of believers.

    I am generally a fan of equal-weight views, and of being deferential to group or expert opinion. However, selection effects like these make deriving the balance of reason from the pattern of belief deeply perplexing.

11. Thanks to Stefan Schubert, Carl Shulman, Amanda MacAskill, Owen Cotton-Barratt and Pablo Stafforini for extensive feedback and advice. Their kind assistance should not be construed as either endorsement endorsement of the content, nor responsibility for any errors.

I grew up in an atheistic household.

Almost needless to say, I was relatively hostile towards religion for most of my early life.  A few things changed that.

First, the apology of a pastor.  A friend of mine was proselytizing at me, and apparently discussed it with his pastor; the pastor apologized to my parents, and explained to my friend he shouldn't be trying to convert people.  My friend apologized to me after considering the matter.  We stayed friends for a little while afterwards, although I left that school, and we lost contact.

I think that was around the time that I realized that religion is, in addition to being a belief system, a way of life, and not necessarily a bad one.

The next was actually South Park's Mormonism episode, which pointed out that a belief system could be desirable on the merits of the way of life it represented, even if the beliefs themselves are stupid.  This tied into Douglas Adam's comment on Feng Shui, that "...if you disregard for a moment the explanation that's actually offered for it, it may be there is something interesting going on" - which is to say, the explanation for the belief is not necessarily the -reason- for the belief, and that stupid beliefs may actually have something useful to offer - which then requires us to ask whether the beliefs are, in fact, stupid.

Which is to say, beliefs may be epistemically irrational while being instrumentally rational.

The next peace I made with belief actually came from quantum physics, and reading about how there were several disparate and apparently contradictory mathematical systems, which all predicted the same thing.  It later transpired that they could all be generalized into the same mathematical system, but I hadn't read that far before the isomorphic nature of truth occurred to me; you can have multiple contradictory interpretations of the same evidence that all predict the same thing.

Up to this point, however, I still regarded beliefs as irrational, at least on an epistemological basis.

The next peace came from experiences living in a house that would have convinced most people that ghosts are real, which I have previously written about here.  I think there are probably good explanations for every individual experience even if I don't know them, but am still somewhat flummoxed by the fact that almost all the bizarre experiences of my life all revolve around the same physical location.  I don't know if I would accept money to live in that house again, which I guess means that I wouldn't put money on the bet that there wasn't something fundamentally odd about the house itself - a quality of the house which I think the term "haunted" accurately conveys, even if its implications are incorrect.

If an AI in a first person shooter dies every time it walks into a green room, and experiences great disutility for death, how many times must it walk into a green room before it decides not to do that anymore?  I'm reasonably confident on a rational level that there was nothing inherently unnatural about that house, nothing beyond explanation, but I still won't "walk into the green room."

That was the point at which I concluded that beliefs can be -rational-.  Disregard for a moment the explanation that's actually offered for them, and just accept the notion that there may be something interesting going on underneath the surface.

If we were to hold scientific beliefs to the same standard we hold religious beliefs - holding the explanation responsible rather than the predictions - scientific beliefs really don't come off looking that good.  The sun isn't the center of the universe; some have called this theory "less wrong" than an earth-centric model of the universe, but that's because the -predictions- are better; the explanation itself is still completely, 100% wrong.

Likewise, if we hold religious beliefs to the same standard we hold scientific beliefs - holding the predictions responsible rather than the explanations - religious beliefs might just come off better than we'd expect.

Below are some notes that I took while trying to understanding what exactly Systems theory is all about.

Winter is coming, and so is Solstice season. There'll be large rationality-centric-or-adjaecent events in NYC, the Bay Area, and Seattle (and possibly other places - if you're interested in running a Solstice event or learning what that involves, send me a PM). In NYC, there'll be a general megameetup throughout the weekend, for people who want to stay through Sunday afternoon, and if you're interested in shared housing you can fill out this form.

The NYC Solstice isn't running a kickstarter this year, but I'll need to pay for the venue by November 19th ($6125). So if you are planning on coming it's helpful to purchase tickets sooner rather than later. (Or preorder the next album or 2016 Book of Traditions, if you can't attend but want to support the event).

-

I've been thinking for the past couple years about the Solstice as a memetic payload.

The Secular Solstice is a (largely Less Wrong inspired) winter holiday, celebrating how humanity faced the darkest season and transformed it into a festival of light. It celebrates science and civilization. It honors the past, revels in the present and promises to carry our torch forward into the future.

For the first 2-3 years, I had a fair amount of influences over the Solstices held in Boston and San Francisco, as well as the one I run in NYC. Even then, the holiday has evolved in ways I didn't quite predict. This has happened both because different communities took them in somewhat different directions, and because (even in the events I run myself), factors come into play that shaped it. Which musicians are available to perform, and how does their stage presence affect the event? Which people from which communities will want to attend, and how will their energy affect things? Which jokes will they laugh at? What will they find poignant?

On top of that, I'm deliberately trying to spread the Solstice to a larger audience. Within a couple years, if I succeed, more of the Solstice will be outside of my control than within it. 

Is it possible to steer a cultural artifact into the future, even after you let go of the reins? How? Would you want to?

In this post, I lay out my current thoughts on this matter. I am interested in feedback, collaboration and criticism.

Lessons from History?

(Epistemic status: I have not really fact checked this. I wouldn't be surprised if the example turned out to be false, but I think it illustrates an interesting point regardless of whether it's true)

Last year after Solstice, I was speaking with a rationalist friend with a Jewish background. He made an observation. I lack the historical background to know if this is exactly accurate (feel free to weigh in on the comments), but his notion was as follows:

Judaism has influenced the world in various direct ways. But a huge portion of its influence (perhaps the majority) has been indirectly through Christianity. Christianity began with a few ideas it took from Judaism that were relatively rare. Monotheism is one example. The notion that you can turn to the Bible for historical and theological truth is another.

But buried in that second point is something perhaps more important: religious truth is not found in the words of your tribal leaders and priests. It's found in a book. The book contains the facts-of-the-matter. And while you can argue cleverly about the book's contents, you can't disregard it entirely.

Empiricists may get extremely frustrated with creationists, for refusing to look outside their book for answers (instead of the natural world). But there was a point where the fact of the matter lay entirely in "what the priests/ruler said" as opposed to "what the book said". 

In this view, Judaism's primary memetic success is in helping to seed the idea of scholarship, and a culture of argument and discussion.

I suspect this story is simplified, but these two points seem meaningful: a memeplex's greatest impact may be indirect, and may not have much to do with the attributes that are most salient on first glance to a layman.

 

Simplicity

So far, I've deliberately encouraged people to experiment with the Solstice. Real rituals evolve in the wild, and adapt to the needs of their community. And a major risk of ritual is that it becomes ossified, turning either hollow or dangerous. But if a ritual is designed to be mutable, what gives it it's identity? What separates a Secular Solstice from a generic humanist winter holiday?

The simplest, most salient and most fun aspects of a ritual will probably spread the fastest and farthest. If I had to sum up the Solstice in nine words, they would be:

Light. Darkness. Light.
Past. Present. Future.
Humanity. Science. Civilization.

I suspect that without any special effort on my part (assuming I keep promoting the event but don't put special effort into steering its direction), those 9 pieces would remain a focus of the event, even if groups I never talk to adopt it for themselves.

The most iconic image of the Solstice is the Candelit story. At the apex of the event, when all lights but a single candle have been extinguished, somebody tells a story that feels personal, visceral. It reminds us that this world can be unfair, but that we are not alone, and we have each other. And then the candle is blown out, and we stand in the absolute darkness together.

If any piece of the Solstice survives, it'll be that moment.

If that were all that survived, I think that'd be valuable. But it'd also be leaving 90%+ of the potential value of the Solstice on the table.

Complex Value

There are several pieces of the Solstice that are subtle and important. There are also pieces of it that currently exist that should probably be tapered down, or adjusted to become more useful. Each of them warrants a fairly comprehensive post of its own. A rough overview of topics to explore:

Atheism.
Rationality.
Death.
Humanism.
Transhumanism.
Existential Risk.
The Here and Now.
The Distant Future.

My thoughts about each of these are fairly complex. In the coming weeks I'll dive into each of them. The next post, discussing Atheism, Rationality and Death, is here.

New essay summarizing some of my latest thoughts on AI safety, ~3500 words. I explain why I think that some of the thought experiments that have previously been used to illustrate the dangers of AI are flawed and should be used very cautiously, why I'm less worried about the dangers of AI than I used to be, and what are some of the remaining reasons for why I do continue to be somewhat worried.

Backcover celebrity endorsement: "Thanks, Kaj, for a very nice write-up. It feels good to be discussing actually meaningful issues regarding AI safety. This is a big contrast to discussions I've had in the past with MIRI folks on AI safety, wherein they have generally tried to direct the conversation toward bizarre, pointless irrelevancies like "the values that would be held by a randomly selected mind", or "AIs with superhuman intelligence making retarded judgments" (like tiling the universe with paperclips to make humans happy), and so forth.... Now OTOH, we are actually discussing things of some potential practical meaning ;p ..." -- Ben Goertzel

FHI is accepting applications for a two-year position as a full-time Research Project Manager. Responsibilities will include coordinating, monitoring, and developing FHI’s activities, seeking funding, organizing workshops and conferences, and effectively communicating FHI’s research. The Research Program Manager will also be expected to work in collaboration with Professor Nick Bostrom, and other researchers, to advance their research agendas, and will additionally be expected to produce reports for government, industry, and other relevant organizations. 

Applicants will be familiar with existing research and literature in the field and have excellent communication skills, including the ability to write for publication. He or she will have experience of independently managing a research project and of contributing to large policy-relevant reports. Previous professional experience working for non-profit organisations, experience with effectiv altruism, and a network in the relevant fields associated with existential risk may be an advantage, but are not essential. 

To apply please go to https://www.recruit.ox.ac.uk and enter vacancy #124775 (it is also possible to find the job by searching choosing “Philosophy Faculty” from the department options). The deadline is noon UK time on 29 August. To stay up to date on job opportunities at the Future of Humanity Institute, please sign up for updates on our vacancies newsletter at https://www.fhi.ox.ac.uk/vacancies/.