This post presents thoughts on the Singularity Institute from Holden Karnofsky, Co-Executive Director of GiveWell. Note: Luke Muehlhauser, the Executive Director of the Singularity Institute, reviewed a draft of this post, and commented: "I do generally agree that your complaints are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI). I am working to address both categories of issues." I take Luke's comment to be a significant mark in SI's favor, because it indicates an explicit recognition of the problems I raise, and thus increases my estimate of the likelihood that SI will work to address them.

September 2012 update: responses have been posted by Luke and Eliezer (and I have responded in the comments of their posts). I have also added acknowledgements.

The Singularity Institute (SI) is a charity that GiveWell has been repeatedly asked to evaluate. In the past, SI has been outside our scope (as we were focused on specific areas such as international aid). With GiveWell Labs we are open to any giving opportunity, no matter what form and what sector, but we still do not currently plan to recommend SI; given the amount of interest some of our audience has expressed, I feel it is important to explain why. Our views, of course, remain open to change. (Note: I am posting this only to Less Wrong, not to the GiveWell Blog, because I believe that everyone who would be interested in this post will see it here.)

I am currently the GiveWell staff member who has put the most time and effort into engaging with and evaluating SI. Other GiveWell staff currently agree with my bottom-line view that we should not recommend SI, but this does not mean they have engaged with each of my specific arguments. Therefore, while the lack of recommendation of SI is something that GiveWell stands behind, the specific arguments in this post should be attributed only to me, not to GiveWell.

Summary of my views

• The argument advanced by SI for why the work it's doing is beneficial and important seems both wrong and poorly argued to me. My sense at the moment is that the arguments SI is making would, if accepted, increase rather than decrease the risk of an AI-related catastrophe. More
• SI has, or has had, multiple properties that I associate with ineffective organizations, and I do not see any specific evidence that its personnel/organization are well-suited to the tasks it has set for itself. More
• A common argument for giving to SI is that "even an infinitesimal chance that it is right" would be sufficient given the stakes. I have written previously about why I reject this reasoning; in addition, prominent SI representatives seem to reject this particular argument as well (i.e., they believe that one should support SI only if one believes it is a strong organization making strong arguments). More
• My sense is that at this point, given SI's current financial state, withholding funds from SI is likely better for its mission than donating to it. (I would not take this view to the furthest extreme; the argument that SI should have some funding seems stronger to me than the argument that it should have as much as it currently has.)
• I find existential risk reduction to be a fairly promising area for philanthropy, and plan to investigate it further. More
• There are many things that could happen that would cause me to revise my view on SI. However, I do not plan to respond to all comment responses to this post. (Given the volume of responses we may receive, I may not be able to even read all the comments on this post.) I do not believe these two statements are inconsistent, and I lay out paths for getting me to change my mind that are likely to work better than posting comments. (Of course I encourage people to post comments; I'm just noting in advance that this action, alone, doesn't guarantee that I will consider your argument.) More

Intent of this post

I did not write this post with the purpose of "hurting" SI. Rather, I wrote it in the hopes that one of these three things (or some combination) will happen:

1. New arguments are raised that cause me to change my mind and recognize SI as an outstanding giving opportunity. If this happens I will likely attempt to raise more money for SI (most likely by discussing it with other GiveWell staff and collectively considering a GiveWell Labs recommendation).
2. SI concedes that my objections are valid and increases its determination to address them. A few years from now, SI is a better organization and more effective in its mission.
3. SI can't or won't make changes, and SI's supporters feel my objections are valid, so SI loses some support, freeing up resources for other approaches to doing good.

Which one of these occurs will hopefully be driven primarily by the merits of the different arguments raised. Because of this, I think that whatever happens as a result of my post will be positive for SI's mission, whether or not it is positive for SI as an organization. I believe that most of SI's supporters and advocates care more about the former than about the latter, and that this attitude is far too rare in the nonprofit world.

Part of the sequence: The Science of Winning at Life

Some have suggested that the Less Wrong community could improve readers' instrumental rationality more effectively if it first caught up with the scientific literature on productivity and self-help, and then enabled readers to deliberately practice self-help skills and apply what they've learned in real life.

I think that's a good idea. My contribution today is a quick overview of scientific self-help: what professionals call "the psychology of adjustment." First I'll review the state of the industry and the scientific literature, and then I'll briefly summarize the scientific data available on three topics in self-help: study methods, productivity, and happiness.

The industry and the literature

As you probably know, much of the self-help industry is a sham, ripe for parody. Most self-help books are written to sell, not to help. Pop psychology may be more myth than fact. As Christopher Buckley (1999) writes, "The more people read [self-help books], the more they think they need them... [it's] more like an addiction than an alliance."

Where can you turn for reliable, empirically-based self-help advice? A few leading therapeutic psychologists (e.g., Albert Ellis, Arnold Lazarus, Martin Seligman) have written self-help books based on decades of research, but even these works tend to give recommendations that are still debated, because they aren't yet part of settled science.

Lifelong self-help researcher Clayton Tucker-Ladd wrote and updated Psychological Self-Help (pdf) over several decades. It's a summary of what scientists do and don't know about self-help methods (as of about 2003), but it's also more than 2,000 pages long, and much of it surveys scientific opinion rather than experimental results, because on many subjects there aren't any experimental results yet. The book is associated with an internet community of people sharing what does and doesn't work for them.

More immediately useful is Richard Wiseman's 59 Seconds. Wiseman is an experimental psychologist and paranormal investigator who gathered together what little self-help research is part of settled science, and put it into a short, fun, and useful Malcolm Gladwell-ish book. The next best popular-level general self-help book is perhaps Martin Seligman's What You Can Change and What You Can't.

Scholarship is an important virtue of rationality, but it can be costly. Its major costs are time and effort. Thus, if you can reduce the time and effort required for scholarship - if you can learn to do scholarship more efficiently - then scholarship will be worth your effort more often than it previously was.

As an autodidact who now consumes whole fields of knowledge in mere weeks, I've developed efficient habits that allow me to research topics quickly. I'll share my research habits with you now.

Review articles and textbooks are king

My first task is to find scholarly review (or 'survey') articles on my chosen topic from the past five years (the more recent, the better). A good review article provides:

1. An overview of the subject matter of the field and the terms being used (for scholarly googling later).
2. An overview of the open and solved problems in the field, and which researchers are working on them.
3. Pointers to the key studies that give researchers their current understanding of the topic.

If you can find a recent scholarly edited volume of review articles on the topic, then you've hit the jackpot. (Edited volumes are better than single-author volumes, because when starting out you want to avoid reading only one particular researcher's perspective.) Examples from my own research of just this year include:

• Affective neuroscience: Pleasures of the Brain (2009)
• Neuroeconomics: Decision Making and the Brain (2008)
• Dual process theories of psychology: In Two Minds (2009)
• Intuition and unconscious learning: Intuition in Judgment and Decision Making (2007)
• Goals: The Psychology of Goals (2009)
• Catastrophic risks: Global Catastrophic Risks (2008)

If the field is large enough, there may exist an edited 'Handbook' on the subject, which is basically just a very large scholarly edited volume of review articles. Examples: Oxford Handbook of Evolutionary Psychology (2007), Oxford Handbook of Positive Psychology (2009), Oxford Handbook of Philosophy and Neuroscience (2009), Handbook of Developmental Cognitive Neuroscience (2008), Oxford Handbook of Neuroethics (2011), Handbook of Relationship Intitiation (2008), and Handbook of Implicit Social Cognition (2010). For the humanities, see the Blackwell Companions and Cambridge Companions.

If your questions are basic enough, a recent entry-level textbook on the subject may be just as good. Textbooks are basically book-length review articles written for undergrads. Textbooks I purchased this year include:

• Evolutionary Psychology: The New Science of Mind, 4th edition (2011)
• Artificial Intelligence: A Modern Approach, 3rd edition (2009)
• Psychology Applied to Modern Life, 10th edition (2011)
• Psychology, 9th edition (2009)

Use Google Books and Amazon's 'Look Inside' feature to see if the books appear to be of high quality, and likely to answer the questions you have. Also check the textbook recommendations here. You can save money by checking Library Genesis and library.nu for a PDF copy first, or by buying used books, or by buying ebook versions from Amazon, B&N, or Google.

(Continued from previous post:  "Why truth?   And...")

A bias is a certain kind of obstacle to our goal of obtaining truth - its character as an "obstacle" stems from this goal of truth - but there are many obstacles that are not "biases".

"Outside the laboratory, scientists are no wiser than anyone else."  Sometimes this proverb is spoken by scientists, humbly, sadly, to remind themselves of their own fallibility.  Sometimes this proverb is said for rather less praiseworthy reasons, to devalue unwanted expert advice.  Is the proverb true?  Probably not in an absolute sense.  It seems much too pessimistic to say that scientists are literally no wiser than average, that there is literally zero correlation.

But the proverb does appear true to some degree, and I propose that we should be very disturbed by this fact.  We should not sigh, and shake our heads sadly.  Rather we should sit bolt upright in alarm.  Why?  Well, suppose that an apprentice shepherd is laboriously trained to count sheep, as they pass in and out of a fold.  Thus the shepherd knows when all the sheep have left, and when all the sheep have returned.  Then you give the shepherd a few apples, and say:  "How many apples?"  But the shepherd stares at you blankly, because they weren't trained to count apples - just sheep.  You would probably suspect that the shepherd didn't understand counting very well.

Now suppose we discover that a Ph.D. economist buys a lottery ticket every week.  We have to ask ourselves:  Does this person really understand expected utility, on a gut level?  Or have they just been trained to perform certain algebra tricks?

Followup to:  Guessing the Teacher's Password, Artificial Addition

A classic paper by Drew McDermott, "Artificial Intelligence Meets Natural Stupidity", criticized AI programs that would try to represent notions like happiness is a state of mind using a semantic network:

STATE-OF-MIND
     ^
     | IS-A
     |
HAPPINESS

And of course there's nothing inside the "HAPPINESS" node; it's just a naked LISP token with a suggestive English name.

So, McDermott says, "A good test for the disciplined programmer is to try using gensyms in key places and see if he still admires his system.  For example, if STATE-OF-MIND is renamed G1073..." then we would have IS-A(HAPPINESS, G1073) "which looks much more dubious."

Or as I would slightly rephrase the idea:  If you substituted randomized symbols for all the suggestive English names, you would be completely unable to figure out what G1071(G1072, 1073) meant.  Was the AI program meant to represent hamburgers?  Apples?  Happiness?  Who knows?  If you delete the suggestive English names, they don't grow back.

Followup to:  Tsuyoku Naritai, Circular Altruism

In the gestalt of (ahem) Japanese fiction, one finds this oft-repeated motif:  Power comes from having something to protect.

I'm not just talking about superheroes that power up when a friend is threatened, the way it works in Western fiction.  In the Japanese version it runs deeper than that.

In the X saga it's explicitly stated that each of the good guys draw their power from having someone—one person—who they want to protect.  Who?  That question is part of X's plot—the "most precious person" isn't always who we think.  But if that person is killed, or hurt in the wrong way, the protector loses their power—not so much from magical backlash, as from simple despair.  This isn't something that happens once per week per good guy, the way it would work in a Western comic.  It's equivalent to being Killed Off For Real—taken off the game board.

The way it works in Western superhero comics is that the good guy gets bitten by a radioactive spider; and then he needs something to do with his powers, to keep him busy, so he decides to fight crime.  And then Western superheroes are always whining about how much time their superhero duties take up, and how they'd rather be ordinary mortals so they could go fishing or something.

Similarly, in Western real life, unhappy people are told that they need a "purpose in life", so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes.  You should be careful not to pick something too expensive, though.

In Western comics, the magic comes first, then the purpose:  Acquire amazing powers, decide to protect the innocent.  In Japanese fiction, often, it works the other way around.

Of course I'm not saying all this to generalize from fictional evidence. But I want to convey a concept whose deceptively close Western analogue is not what I mean.

I have touched before on the idea that a rationalist must have something they value more than "rationality":  The Art must have a purpose other than itself, or it collapses into infinite recursion.  But do not mistake me, and think I am advocating that rationalists should pick out a nice altruistic cause, by way of having something to do, because rationality isn't all that important by itself.  No.  I am asking:  Where do rationalists come from?  How do we acquire our powers? 

Followup to:  Fake Selfishness

God, say the religious fundamentalists, is the source of all morality; there can be no morality without a Judge who rewards and punishes.  If we did not fear hell and yearn for heaven, then what would stop people from murdering each other left and right?

Suppose Omega makes a credible threat that if you ever step inside a bathroom between 7AM and 10AM in the morning, he'll kill you. Would you be panicked by the prospect of Omega withdrawing his threat?  Would you cower in existential terror and cry:  "If Omega withdraws his threat, then what's to keep me from going to the bathroom?"  No; you'd probably be quite relieved at your increased opportunity to, ahem, relieve yourself.

Which is to say:  The very fact that a religious person would be afraid of God withdrawing Its threat to punish them for committing murder, shows that they have a revulsion of murder which is independent of whether God punishes murder or not.  If they had no sense that murder was wrong independently of divine retribution, the prospect of God not punishing murder would be no more existentially horrifying than the prospect of God not punishing sneezing.