I recently wrote an essay about AI risk, targeted at other academics:

Long-Term and Short-Term Challenges to Ensuring the Safety of AI Systems

I think it might be interesting to some of you, so I am sharing it here. I would appreciate any feedback any of you have, especially from others who do AI / machine learning research.

Recently Luke Muelhauser posed the question, “Can noise have power?”, which basically asks whether randomization can ever be useful, or whether for every randomized algorithm there is a better deterministic algorithm. This question was posed in response to a debate between Eliezer Yudkowsky and Scott Aaronson, in which Eliezer contends that randomness (or, as he calls it, noise) can't ever be helpful, and Scott takes the opposing view. My goal in this essay is to present my own views on this question, as I feel many of them have not yet been brought up in discussion.

I'll spare the reader some suspense and say that I basically agree with Scott. I also don't think – as some others have suggested – that this debate can be chalked up to a dispute about the meaning of words. I really do think that Scott is getting at something important in the points he makes, which may be underappreciated by those without a background in a field such as learning theory or game theory.

Before I start, I'd like to point out that this is really a debate about Bayesianism in disguise. Suppose that you're a Bayesian, and you have a posterior distribution over the world, and a utility function, and you are contemplating two actions A and B, with expected utilities U(A) and U(B). Then randomly picking between A and B will have expected utility , and so in particular at least one of A and B must have higher expected utility than randomizing between them. One can extend this argument to show that, for a Bayesian, the best strategy is always deterministic. Scott in fact acknowledges this point, although in slightly different language:

“Randomness provably never helps in average-case complexity (i.e., where you fix the probability distribution over inputs) -- since given any ensemble of strategies, by convexity there must be at least one deterministic strategy in the ensemble that does at least as well as the average.” -Scott Aaronson

I think this point is pretty iron-clad and I certainly don't wish to argue against it. Instead, I'd like to present several examples of scenarios where I will argue that randomness clearly is useful and necessary, and use this to argue that, at least in these scenarios, one should abandon a fully Bayesian stance. At the meta level, this essay is therefore an argument in favor of maintaining multiple paradigms (in the Kuhnian sense) with which to solve problems.

I will make four separate arguments, paralleling the four main ways in which one might argue for the adoption or dismissal of a paradigm:

1. Randomization is an appropriate tool in many concrete decision-making problems (game theory and Nash equilibria, indivisible goods, randomized controlled trials).

2. Worst case analyses (which typically lead to randomization) are often important from an engineering design perspective (modularity of software).

3. Random algorithms have important theoretical properties not shared by deterministic algorithms (P vs. BPP).

4. Thinking in terms of randomized constructions has solved many problems that would have been difficult or impossible without this perspective (probabilistic method, sampling algorithms).

Recently Ben Kuhn wrote a critique of effective altruism. I'm glad to see such self-examination taking place, but I'm also concerned that the essay did not attack some of the most serious issues I see in the effective altruist movement, so I've decided to write my own critique. Due to time constraints, this critique is short and incomplete. I've tried to bring up arguments that would make people feel uncomfortable and defensive; hopefully I've succeeded.

Briefly, here are some of the major issues I have with the effective altruism movement as it currently stands:

• Over-focus on “tried and true” and “default” options, which may both reduce actual impact and decrease exploration of new potentially high-value opportunities.

• Over-confident claims coupled with insufficient background research.

• Over-reliance on a small set of tools for assessing opportunities, which lead many to underestimate the value of things such as “flow-through” effects.

The common theme here is a subtle underlying message that simple, shallow analyses can allow one to make high-impact career and giving choices, and divest one of the need to dig further. I doubt that anyone explicitly believes this, but I do believe that this theme comes out implicitly both in arguments people make and in actions people take.

Lest this essay give a mistaken impression to the casual reader, I should note that there are many examplary effective altruists who I feel are mostly immune to the issues above; for instance, the GiveWell blog does a very good job of warning against the first and third points above, and I would recommend anyone who isn't already to subscribe to it (and there are other examples that I'm failing to mention). But for the purposes of this essay, I will ignore this fact except for the current caveat.

Over-focus on "tried and true" options

It seems to me that the effective altruist movement over-focuses on “tried and true” options, both in giving opportunities and in career paths. Perhaps the biggest example of this is the prevalence of “earning to give”. While this is certainly an admirable option, it should be considered as a baseline to improve upon, not a definitive answer.

The biggest issue with the “earning to give” path is that careers in finance and software (the two most common avenues for this) are incredibly straight-forward and secure. The two things that finance and software have in common is that there is a well-defined application process similar to the one for undergraduate admissions, and given reasonable job performance one will continue to be given promotions and raises (this probably entails working hard, but the end result is still rarely in doubt). One also gets a constant source of extrinsic positive reinforcement from the money they earn. Why do I call these things an “issue”? Because I think that these attributes encourage people to pursue these paths without looking for less obvious, less certain, but ultimately better paths. One in six Yale graduates go into finance and consulting, seemingly due to the simplicity of applying and the easy supply of extrinsic motivation. My intuition is that this ratio is higher than an optimal society would have, even if such people commonly gave generously (and it is certainly much higher than the number of people who enter college planning to pursue such paths).

Contrast this with, for instance, working at a start-up. Most start-ups are low-impact, but it is undeniable that at least some have been extraordinarily high-impact, so this seems like an area that effective altruists should be considering strongly. Why aren't there more of us at 23&me, or Coursera, or Quora, or Stripe? I think it is because these opportunities are less obvious and take more work to find, once you start working it often isn't clear whether what you're doing will have a positive impact or not, and your future job security is massively uncertain. There are few sources of extrinsic motivation in such a career: perhaps moreso at one of the companies mentioned above, which are reasonably established and have customers, but what about the 4-person start-up teams working in a warehouse somewhere? Some of them will go on to do great things but right now their lives must be full of anxiousness and uncertainty.

I don't mean to fetishize start-ups. They are just one well-known example of a potentially high-value career path that, to me, seems underexplored within the EA movement. I would argue (perhaps self-servingly) that academia is another example of such a path, with similar psychological obstacles: every 5 years or so you have the opportunity to get kicked out (e.g. applying for faculty jobs, and being up for tenure), you need to relocate regularly, few people will read your work and even fewer will praise it, and it won't be clear whether it had a positive impact until many years down the road. And beyond the “obvious” alternatives of start-ups and academia, what of the paths that haven't been created yet? GiveWell was revolutionary when it came about. Who will be the next GiveWell? And by this I don't mean the next charity evaluator, but the next set of people who fundamentally alter how we view altruism.

Over-confident claims coupled with insufficient background research

The history of effective altruism is littered with over-confident claims, many of which have later turned out to be false. In 2009, Peter Singer claimed that you could save a life for $200 (and many others repeated his claim). While the number was already questionable at the time, by 2011 we discovered that the number was completely off. Now new numbers were thrown around: from numbers still in the hundreds of dollars (GWWC's estimate for SCI, which was later shown to be flawed) up to $1600 (GiveWell's estimate for AMF, which GiveWell itself expected to go up, and which indeed did go up). These numbers were often cited without caveats, as well as other claims such as that the effectiveness of charities can vary by a factor of 1,000. How many people citing these numbers understood the process that generated them, or the high degree of uncertainty surrounding them, or the inaccuracy of past estimates? How many would have pointed out that saying that charities vary by a factor of 1,000 in effectiveness is by itself not very helpful, and is more a statement about how bad the bottom end is than how good the top end is?

More problematic than the careless bandying of numbers is the tendency toward not doing strong background research. A common pattern I see is: an effective altruist makes a bold claim, then when pressed on it offers a heuristic justification together with the claim that “estimation is the best we have”. This sort of argument acts as a conversation-stopper (and can also be quite annoying, which may be part of what drives some people away from effective altruism). In many of these cases, there are relatively easy opportunities to do background reading to further educate oneself about the claim being made. It can appear to an outside observer as though people are opting for the fun, easy activity (speculation) rather than the harder and more worthwhile activity (research). Again, I'm not claiming that this is people's explicit thought process, but it does seem to be what ends up happening.

Why haven't more EAs signed up for a course on global security, or tried to understand how DARPA funds projects, or learned about third-world health? I've heard claims that this would be too time-consuming relative to the value it provides, but this seems like a poor excuse if we want to be taken seriously as a movement (or even just want to reach consistently accurate conclusions about the world).

Over-reliance on a small set of tools

Effective altruists tend to have a lot of interest in quantitative estimates. We want to know what the best thing to do is, and we want a numerical value. This causes us to rely on scientific studies, economic reports, and Fermi estimates. It can cause us to underweight things like the competence of a particular organization, the strength of the people involved, and other “intangibles” (which are often not actually intangible but simply difficult to assign a number to). It also can cause us to over-focus on money as a unit of altruism, while often-times “it isn't about the money”: it's about doing the groundwork that no one is doing, or finding the opportunity that no one has found yet.

Quantitative estimates often also tend to ignore flow-through effects: effects which are an indirect, rather than direct, result of an action (such as decreased disease in the third world contributing in the long run to increased global security). These effects are difficult to quantify but human and cultural intuition can do a reasonable job of taking them into account. As such, I often worry that effective altruists may actually be less effective than “normal” altruists. (One can point to all sorts of examples of farcical charities to claim that regular altruism sucks, but this misses the point that there are also amazing organizations out there, such as the Simons Foundation or HHMI, which are doing enormous amounts of good despite not subscribing to the EA philosophy.)

What's particularly worrisome is that even if we were less effective than normal altruists, we would probably still end up looking better by our own standards, which explicitly fail to account for the ways in which normal altruists might outperform us (see above). This is a problem with any paradigm, but the fact that the effective altruist community is small and insular and relies heavily on its paradigm makes us far more susceptible to it.

Overview

The basic observation is that, if we think of life as an optimization problem, then redefining the search space is much more important than making local optimizations; as a fact of human psychology it's hard to consciously focus on both; but we can implicitly get away with doing both by creating mental triggers for when local optimizations are likely to be particularly effective to think about, and by structuring things so that many local optimizations get made automatically.

Introduction

If you have been to one of the Rationality Minicamps or certain other CFAR events, you may have had the privilege to attend one of Anna Salamon's excellent classes on microeconomics (despite the title of the post, I am being sincere here; you really should attend them if you haven't already). There is too much content to briefly summarize, but essentially "microeconomics" in this context means applying basic microeconomic concepts like marginal value, value of information, etc. to everyday life. For example, if you spend 30 seconds brushing your teeth each day, then spending five minutes to think of something else to do at the same time (like stretching) will save you 3 hours a year, which is a great investment! (There are some caveats to this calculation, but I'm glossing over them as they aren't relevant to the post.)

And indeed, spending 5 minutes (once) to save 3 hours (every year) is almost tautologically a good investment. Now that I've brought up this example, and assuming you value your time, you should probably actually go through this exercise (or just use the stretching suggestion).

The Problem

I intend to argue against something similar to this but subtly different. Basically, while any given trade such as the one above is good, I think it is a mistake to systematically search for such trades. Note that I also don't want to argue that you should never search for such trades. If you're about to buy a car you should almost certainly put a lot of microeconomic optimization into it, and if you can find things that improve your overall work efficiency substantially, then you are winning big-time. But I worry that, sometimes, the wrong lesson is drawn from these microeconomics examples (or cognitive bias examples, or any other rationality skill), namely that X is suboptimal by default and we should go out looking for places to optimize X.

The reason I think this is wrong is because it aims much too low --- if you really want to save the world, then your average thought needs to be good enough to save two human lives [source: the average human lives only 3 billion seconds]. Even if each individual optimization you make ends up adding to the amount of time you have, the overall process of concentrating your attention on such optimizations makes you less likely to think other thoughts that would be far more valuable. Perhaps another way of putting this is that, even if each small optimization helps you a little bit, the time it takes to think up such optimizations actually makes you lose out --- however, I don't think this is actually it, I think it has more to do with forming mental habits, where you want to form the mental habit of making huge optimizations rather than small optimizations.

The Solution

What I think people should be more concerned with than micro is what I'll refer to as macro --- the overall structure of the search space (in this case the structure of your life and how you think) --- as opposed to making local optimizations within a fixed structure. For instance, becoming an atheist; or realizing that social skills are both trainable and highly instrumentally useful; or learning to visualize the steps towards a goal; or learning to code; or finding a group of allies that you didn't previously realize existed; these are all examples of what I'd call "macro" optimizations that are the sorts of things we should be looking for. (I should note that a lot of "macro" skills were also covered in Anna's microeconomics units.)

I also continue to think that there is a clear place for micro-level skills, as well. The key is to incorporate them into your thought process, both at the level of creating triggers to explicitly call micro-level optimization routines when they are likely to be helpful, and at the level of restructuring your thought process to automatically be more likely to make good decisions by default. For instance, the lesson I drew from Eliezer's posts on cognitive biases were not that we should go learn about all the different cognitive biases, but that we should develop habits of thought that will automatically notice and decrease the effects of such biases. Then, for a couple of the more pernicious ones like trivial inconveniences, I further added specific alarm bells in my head to watch out for those, but only because I noticed that avoiding trivial inconveniences was routinely harming me.

Conclusion

I'm not sure how good of a job I've done of explaining what I wanted to (it's still not entirely clear in my own head), so I invite your thoughts and feedback. I'd be particularly grateful if someone wiser than me (I'm looking at you, Critch / Wei / Yvain) could figure out what this post was trying to say and then write that instead!

(Note: this is cross-posted from my blog and also available in pdf here.)

If you are a newly initiated student into the field of machine learning, it won't be long before you start hearing the words "Bayesian" and "frequentist" thrown around. Many people around you probably have strong opinions on which is the "right" way to do statistics, and within a year you've probably developed your own strong opinions (which are suspiciously similar to those of the people around you, despite there being a much greater variance of opinion between different labs). In fact, now that the year is 2012 the majority of new graduate students are being raised as Bayesians (at least in the U.S.) with frequentists thought of as stodgy emeritus professors stuck in their ways.

If you are like me, the preceding set of facts will make you very uneasy. They will make you uneasy because simple pattern-matching -- the strength of people's opinions, the reliability with which these opinions split along age boundaries and lab boundaries, and the ridicule that each side levels at the other camp – makes the "Bayesians vs. frequentists" debate look far more like politics than like scholarly discourse. Of course, that alone does not necessarily prove anything; these disconcerting similarities could just be coincidences that I happened to cherry-pick.

My next point, then, is that we are right to be uneasy, because such debate makes us less likely to evaluate the strengths and weaknesses of both approaches in good faith. This essay is a push against that --- I summarize the justifications for Bayesian methods and where they fall short, show how frequentist approaches can fill in some of their shortcomings, and then present my personal (though probably woefully under-informed) guidelines for choosing which type of approach to use.

Before doing any of this, though, a bit of background is in order...

1. Background on Bayesians and Frequentists

1.1. Three Levels of Argument

As Andrew Critch [6] insightfully points out, the Bayesians vs. frequentists debate is really three debates at once, centering around one or more of the following arguments:

1. Whether to interpret subjective beliefs as probabilities
2. Whether to interpret probabilities as subjective beliefs (as opposed to asymptotic frequencies)
3. Whether a Bayesian or frequentist algorithm is better suited to solving a particular problem.

Given my own research interests, I will add a fourth argument:

4. Whether Bayesian or frequentist techniques are better suited to engineering an artificial intelligence.

Andrew Gelman [9] has his own well-written essay on the subject, where he expands on these distinctions and presents his own more nuanced view.

Why are these arguments so commonly conflated? I'm not entirely sure; I would guess it is for historical reasons but I have so far been unable to find said historical reasons. Whatever the reasons, what this boils down to in the present day is that people often form opinions on 1. and 2., which then influence their answers to 3. and 4. This is not good, since 1. and 2. are philosophical in nature and difficult to resolve correctly, whereas 3. and 4. are often much easier to resolve and extremely important to resolve correctly in practice. Let me re-iterate: the Bayes vs. frequentist discussion should center on the practical employment of the two methods, or, if epistemology must be discussed, it should be clearly separated from the day-to-day practical decisions. Aside from the difficulties with correctly deciding epistemology, the relationship between generic epistemology and specific practices in cutting-edge statistical research is only via a long causal chain, and it should be completely unsurprising if Bayesian epistemology leads to the employment of frequentist tools or vice versa.

Audio books have been discussed a bit before, but I never saw a list of recommendations. What books are good in an audio format (especially ones that can be listened to while driving)? I commute 45 minutes a day and would like to put that time to good use.

ETA: I'm mostly interested in non-fiction (goal here is to learn useful stuff, as opposed to entertainment).

I recently came out against paleo in the open thread, and realized that I probably haven't yet heard the strongest arguments in favor of a paleo diet. So, what are said arguments?

EDIT: Or more generally, why should I eat less carbohydrates and more protein / fat?

enough people have reported this that i wanted to make it publicly known. on my phone so will let others provide more detail

In September I will be moving to Mountain View, CA together with a friend of mine from MIT. It turns out that the quality/cost ratio increases noticeably with the number of people living together, so we are looking for one or more additional roommates. All things being equal, I would much rather live with other rationalists, which is why I'm posting this on LessWrong.

About us

My name is Jacob, and my roommate's name is Jonathan. We both recently graduated from MIT (me with a bachelor's in mathematics, him with a master's in electrical engineering). I am going to graduate school in machine learning at Stanford; Jonathan works at Synaptics (the company that makes touch sensors).

You can find approximately four-month-outdated information about me at my old MIT website. I've been awarded both the Hertz and NSF Fellowships, which means that I have a guaranteed source of income for the next five years regardless of any external factors like my adviser's ability to pay for me. I teach for SPARC (CFAR's high school program) and am very interested in building up the rationalist community in the south bay.

Reasons you should live with us

• we both have steady sources of income and are on highly successful career tracks
• your behavior is strongly affected by the culture you live in; living with other rationalists will make you more rational
• we both value open communication and are difficult to offend, which makes conflict resolution much easier
• be at the center of exciting developments: I am working directly on important problems in AI and rationalist outreach, and know an embarrassingly large amount of math / computer science, even by LessWrong standards
• I know a lot about sports and strength training, and am happy to help you out if your goal is to become stronger / more athletic
• I am also first aid and CPR certified, so you are slightly less likely to die if you live with me
• Jonathan is a pretty good cook and would be interested in leading the effort on group dinners and/or teaching some culinary basics.
• Jonathan is good at do-it-yourself electronics (sensors, microcontrollers, FPGAs) and is willing to share experience / expertise
• we are both willing to participate in house-wide life-hacking experiments (N = 3 is much better than N = 1 for data size)

What we are looking for

• interest in building up the rationalist community
• steady source of income
• you are interesting to talk to and instrumentally rational

Contact info

If you might be interested in living with us, send me a PM telling me a little bit about yourself; I'll then give you my e-mail and we can figure out if there is likely to be a good fit (obviously, we will also meet in person before any final decisions are made).

EDIT: One person said via email that they tried and failed to PM me. In case this is a larger issue, my e-mail is jsteinha@csail.mit.edu.