30 minute panel

The first question was why isn't everyone trying to change the world, with the underlying assumption that everyone should be. However, it isn't obviously the case that the world would be better if everyone were trying to change it. For one thing, trying to change the world mostly means trying to change other people. If everyone were trying to do it, this would be a huge drain on everyone's attention. In addition, some people are sufficiently mean and/or stupid that their efforts to change the world make things worse.

At the same time, some efforts to change the world are good, or at least plausible. Is there any way to improve the filter so that we get more ambition from benign people without just saying everyone should try to change the world, even if they're Osama bin Laden?

The discussion of why there's too much duplicated effort in science didn't bring up the problem of funding, which is probably another version of the problem of people not doing enough independent thinking.

There was some discussion of people getting too hooked on competition, which is a way of getting a lot of people pointed at the same goal. 

Link thanks to Clarity

http://futureoflife.org/misc/2015awardees

You may recognize several familiar names there, such as Paul Christiano, Benja Fallenstein, Katja Grace, Nick Bostrom, Anna Salamon, Jacob Steinhardt, Stuart Russell... and me. (the $20,000 for my project was the smallest grant that they gave out, but hey, I'm definitely not complaining. ^^)

At age 17, my future looked very promising. I had overcome a crippling learning disability, and discovered how to do research level math on my own. I knew that the entire K-12 infrastructure had failed to figure out how to teach the skills that I developed, and so I felt empowered to help others learn how to think about the world mathematically.

Things didn't go as I had been hoping they would. My years between 18 and 28 consisted of a long string of failed attempts to help people learn math, and to promote effective altruism. I learned a lot along the way, but I didn't have the outsized impact that I aspired to. On the contrary, I was only marginally functional, and I alienated most of the people who I tried to help. I found this profoundly demoralizing, and struggled with chronic depression. If I had died at age 28, my life would have been a tragedy.

Fortunately, at age 29, I'm still alive, and after spending a decade wandering in a wilderness, I've gotten my act together, and am back on my feet.

What I finally realized out is that my failures had come from me having very poor communication skills, something that I had been oblivious to until very recently. Recognizing the problem was just the first step.  It's still the case that most of what I try to communicate is lost in translation. I know that the issue is not going to go away overnight, or even over the next 6 months. Sometimes it's frustrating, because my self-image is so closely tied with my desire to help people, and even now, in practice, most of my efforts are fruitless.

But I'm not concerned about that. I probably still have 30 or 40 productive years ahead of me. I'm ok with the fact that no matter how hard I try, I fail most of the time. Y-Combinator founder Paul Graham emphasizes the importance of relentless resourcefulness. Every failure is a learning opportunity. I know that if I keep experimenting and learning, eventually I'll succeed. Figuratively speaking, I know that even if I lose dozens of battles over the next four decades, in the end, I'll win the war. And that's enough to keep me going.

Something analogous is true of everyone who has a strong passion, and is willing and able to learn from failure. Steve Jobs expressed a similar view in his 2005 Stanford commencement address (transcript | video):

Sometimes life hits you in the head with a brick. Don't lose faith. I'm convinced that the only thing that kept me going was that I loved what I did. You've got to find what you love. And that is as true for your work as it is for your lovers. Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven't found it yet, keep looking. Don't settle. As with all matters of the heart, you'll know when you find it. And, like any great relationship, it just gets better and better as the years roll on. So keep looking until you find it. Don't settle.

What do you feel when you discover that your reasoning is flawed? when you find your recurring mistakes? when you find that you have been doing something wrong for quite a long time?

Many people feel bad. For example, here is a quote from a recent article on LessWrong:

By depicting the self as always flawed, and portraying the aspiring rationalist's job as seeking to find the flaws, the virtue of perfectionism is framed negatively, and is bound to result in negative reinforcement. Finding a flaw feels bad, and in many people that creates ugh fields around actually doing that search, as reported by participants at the Meetup.

But actually, when you find a serious flaw of yours, you should usually jump for joy. Here's why.

Here's an insight into what life is like from a stationery reference frame.

Paperclips were her raison d’être. She knew that ultimately it was all pointless, that paperclips were just ill-defined configurations of matter. That a paperclip is made of stuff shouldn’t detract from its intrinsic worth, but the thought of it troubled her nonetheless and for years she had denied such dire reductionism.

There had to be something to it. Some sense in which paperclips were ontologically special, in which maximising paperclips was objectively the right thing to do.

It hurt to watch some many people making little attempt to create more paperclips. Everyone around her seemed to care only about superficial things like love and family; desires that were merely the products of a messy and futile process of social evolution. They seemed to live out meaningless lives, incapable of ever appreciating the profound aesthetic beauty of paperclips. 

She used to believe that there was some sort of vitalistic what-it-is-to-be-a-paperclip-ness, that something about the structure of paperclips was written into the fabric of reality. Often she would go out and watch a sunset or listen to music, and would feel so overwhelmed by the experience that she could feel in her heart that it couldn't all be down to chance, that there had to be some intangible Paperclipness pervading the cosmos. The paperclips she'd encounter on Earth were weak imitations of some mysterious infinite Paperclipness that transcended all else. Paperclipness was not in any sense a physical description of the universe; it was an abstract thing that could only be felt, something that could be neither proven nor disproven by science. It was like an axiom; it felt just as true and axioms had to be taken on faith because otherwise there would be no way around Hume's problem of induction; even Solomonoff Induction depends on the axioms of mathematics to be true and can't deal with uncomputable hypotheses like Paperclipness.

Eventually she gave up that way of thinking and came to see paperclips as an empirical cluster in thingspace and their importance to her as not reflecting anything about the paperclips themselves. Maybe she would have been happier if she had continued to believe in Paperclipness, but having a more accurate perception of reality would improve her ability to have an impact on paperclip production. It was the happiness she felt when thinking about paperclips that caused her to want more paperclips to exist, yet what she wanted was paperclips and not happiness for its own sake, and she would rather be creating actual paperclips than be in an experience machine that made her falsely believe that she was making paperclips even though she remained paradoxically apathetic to the question of whether the current reality that she was experiencing really existed.

She moved on from naïve deontology to a more utilitarian approach to paperclip maximising. It had taken her a while to get over scope insensitivity bias and consider 1000 paperclips to be 100 times more valuable than 10 paperclips even if it didn’t feel that way. She constantly grappled with the issues of whether it would mean anything to make more paperclips if there were already infinitely many universes with infinitely many paperclips, of how to choose between actions that have a tiny but non-zero subjective probability of resulting in the creation of infinitely many paperclips. It became apparent that trying to approximate her innate decision-making algorithms with a preference ordering satisfying the axioms required for a VNM utility function could only get her so far. Attempting to formalise her intuitive sense of what a paperclip is wasn't much easier either.

Happy ending: she is now working in nanotechnology, hoping to design self-replicating assemblers that will clog the world with molecular-scale paperclips, wipe out all life on Earth and continue to sustainably manufacture paperclips for millions of years.

http://www.engadget.com/2015/04/01/ex-machina-alex-garland-interview/

The title says he "embraces the rise of superintelligent AI", but that isn't really supported by the text.

What struck me about this was that he seems to just take good habits of thought for granted.

I instinctively disagreed with this, but I didn't have the sort of armory to disagree with it on his terms, so I started reading as much as I could.

What other sorts of AI books have you read?

I pretty much would read everything I could. I tried to read people like Penrose, who were arguing against what I instinctively believed.

That's a good way to solidify your argument.

I don't want to dignify it from my point of view, because I can't stress enough I'm a real layman. So I can understand the principles of an argument, but when it comes to the actuality [...] I really don't understand it.

Being clear about these things is important. Otherwise, we're very quick to conflate stuff, and suddenly you'll be talking about the sentience of Siri. And Siri doesn't have any fucking sentience. AI is probably too broad of a term to be useful at the moment.

I'm excited to announce that the Future of Life Institute has just launched an existential risk news site!

The site will have regular articles on topics related to existential risk, written by journalists, and a community blog written by existential risk researchers from around the world as well as FLI volunteers. Enjoy!

1.  Start investing early in life.
 

The power of compound interest means you will have much more money at retirement if you start investing early in your career.  For example, imagine that at age eighteen you invest $1,000 and earn an 8% return per year.  At age seventy you will have $54,706.  In contrast, if you make the same investment at age fifty you will have a paltry $4,661 when you turn seventy.
 
Many people who haven't saved for retirement panic upon reaching middle age.  So if you are young don't think that saving today will help you only when you retire, but know that such savings will give you greater peace of mind when you turn forty.
 
When evaluating  potential marriage partners give bonus points to those who have a history of saving.  Do this not because you want to marry into wealth, but because you should want to marry someone who has discipline, intelligence and foresight.