This is a short post to say that I have started and am managing a Slack channel for lesswrong.

Slack has only an email-invite option which means that I need an email address for anyone who wants to join.  Send me a PM with your email address if you are interested in joining.

There is a web interface and a mobile app that is better than google hangouts.

If you are interested in joining; consider this one requirement:

• You must be willing to be charitable in your conversations with your fellow lesswrongers.

To be clear; This means (including but not limited to);

• Steelman not strawman of discussion
• Respect of others
• patience

• an aim for productive conversations, to make progress on our lives.
• a brains trust for life-advice in all kinds of areas where, "outsource this decision to others" is an effective strategy.
• collaborative creation of further rationality content
• a safe space for friendly conversation on the internet (a nice place to hang out)
• A more coherent and stronger connected lesswrong
• Development of better ideas and strategies in how to personally improve the world.

Edit - first week of October: http://lesswrong.com/r/discussion/lw/mub/open_thread_oct_5_oct_11_2015/csr3

Edit - 3rd week in october 2015: http://lesswrong.com/r/discussion/lw/mwt/open_thread_oct_26_nov_01_2015/cuq5

Edit - 3rd week in november 2015 http://lesswrong.com/r/discussion/lw/mzz/open_thread_nov_23_nov_29_2015/cwyl

[Many people have been complaining about the lack of new content on LessWrong lately, so I thought I'd cross-post my latest blog post here in discussion. Feel free to critique the content as much as you like, but please do keep in mind that I wrote this for my personal blog and not with LW in mind specifically, so some parts might not be up to LW standards, whereas others might be obvious to everyone here. In other words...well, be gentle]

---------------------------

You know what’s scarier than having enemy soldiers at your border?

Having sleeper agents within your borders.

Enemy soldiers are malevolent, but they are at least visibly malevolent. You can see what they’re doing; you can fight back against them or set up defenses to stop them. Sleeper agents on the other hand are malevolent and invisible. They are a threat and you don’t know that they’re a threat. So when a sleeper agent decides that it’s time to wake up and smell the gunpowder, not only will you be unable to stop them, but they’ll be in a position to do far more damage than a lone soldier ever could. A single well-placed sleeper agent can take down an entire power grid, or bring a key supply route to a grinding halt, or – in the worst case – kill thousands with an act of terrorism, all without the slightest warning.

Okay, so imagine that your country is in wartime, and that a small group of vigilant citizens has uncovered an enemy sleeper cell in your city. They’ve shown you convincing evidence for the existence of the cell, and demonstrated that the cell is actively planning to commit some large-scale act of violence – perhaps not imminently, but certainly in the near-to-mid-future. Worse, the cell seems to have even more nefarious plots in the offing, possibly involving nuclear or biological weapons.

Now imagine that when you go to investigate further, you find to your surprise and frustration that no one seems to be particularly concerned about any of this. Oh sure, they acknowledge that in theory a sleeper cell could do some damage, and that the whole matter is probably worthy of further study. But by and large they just hear you out and then shrug and go about their day. And when you, alarmed, point out that this is not just a theory – that you have proof that a real sleeper cell is actually operating and making plans right now – they still remain remarkably blase. You show them the evidence, but they either don’t find it convincing, or simply misunderstand it at a very basic level (“A wiretap? But sleeper agents use cellphones, and cellphones are wireless!”). Some people listen but dismiss the idea out of hand, claiming that sleeper cell attacks are “something that only happen in the movies”. Strangest of all, at least to your mind, are the people who acknowledge that the evidence is convincing, but say they still aren’t concerned because the cell isn’t planning to commit any acts of violence imminently, and therefore won’t be a threat for a while. In the end, all of your attempts to raise the alarm are to no avail, and you’re left feeling kind of doubly scared – scared first because you know the sleeper cell is out there, plotting some heinous act, and scared second because you know you won’t be able to convince anyone of that fact before it’s too late to do anything about it.

This is roughly how I feel about AI risk.

You see, I think artificial intelligence is probably the most significant existential threat facing humanity right now. This, to put it mildly, is something of a fringe position in most intellectual circles (although that’s becoming less and less true as time goes on), and I’ll grant that it sounds kind of absurd. But regardless of whether or not you think I’m right to be scared of AI, you can imagine how the fact that AI risk is really hard to explain would make me even more scared about it. Threats like nuclear war or an asteroid impact, while terrifying, at least have the virtue of being simple to understand – it’s not exactly hard to sell people on the notion that a 2km hunk of rock colliding with the planet might be a bad thing. As a result people are aware of these threats and take them (sort of) seriously, and various organizations are (sort of) taking steps to stop them.

AI is different, though. AI is more like the sleeper agents I described above – frighteningly invisible. The idea that AI could be a significant risk is not really on many people’s radar at the moment, and worse, it’s an idea that resists attempts to put it on more people’s radar, because it’s so bloody confusing a topic even at the best of times. Our civilization is effectively blind to this threat, and meanwhile AI research is making progress all the time. We’re on the Titanic steaming through the North Atlantic, unaware that there’s an iceberg out there with our name on it – and the captain is ordering full-speed ahead.

(That’s right, not one but two ominous metaphors. Can you see that I’m serious?)

But I’m getting ahead of myself. I should probably back up a bit and explain where I’m coming from.

Artificial intelligence has been in the news lately. In particular, various big names like Elon Musk, Bill Gates, and Stephen Hawking have all been sounding the alarm in regards to AI, describing it as the greatest threat that our species faces in the 21st century. They (and others) think it could spell the end of humanity – Musk said, “If I had to guess what our biggest existential threat is, it’s probably [AI]”, and Gates said, “I…don’t understand why some people are not concerned [about AI]”.

Of course, others are not so convinced – machine learning expert Andrew Ng said that “I don’t work on not turning AI evil today for the same reason I don’t worry about the problem of overpopulation on the planet Mars”.

In this case I happen to agree with the Musks and Gates of the world – I think AI is a tremendous threat that we need to focus much of our attention on it in the future. In fact I’ve thought this for several years, and I’m kind of glad that the big-name intellectuals are finally catching up.

Why do I think this? Well, that’s a complicated subject. It’s a topic I could probably spend a dozen blog posts on and still not get to the bottom of. And maybe I should spend those dozen-or-so blog posts on it at some point – it could be worth it. But for now I’m kind of left with this big inferential gap that I can’t easily cross. It would take a lot of explaining to explain my position in detail. So instead of talking about AI risk per se in this post, I thought I’d go off in a more meta-direction – as I so often do – and talk about philosophical differences in general. I figured if I couldn’t make the case for AI being a threat, I could at least make the case for making the case for AI being a threat.

(If you’re still confused, and still wondering what the whole deal is with this AI risk thing, you can read a not-too-terrible popular introduction to the subject here, or check out Nick Bostrom’s TED Talk on the topic. Bostrom also has a bestselling book out called Superintelligence. The one sentence summary of the problem would be: how do we get a superintelligent entity to want what we want it to want?)

(Trust me, this is much much harder than it sounds)

So: why then am I so meta-concerned about AI risk? After all, based on the previous couple paragraphs it seems like the topic actually has pretty decent awareness: there are popular internet articles and TED talks and celebrity intellectual endorsements and even bestselling books! And it’s true, there’s no doubt that a ton of progress has been made lately. But we still have a very long way to go. If you had seen the same number of online discussions about AI that I’ve seen, you might share my despair. Such discussions are filled with replies that betray a fundamental misunderstanding of the problem at a very basic level. I constantly see people saying things like “Won’t the AI just figure out what we want?”, or “If the AI gets dangerous why can’t we just unplug it?”, or “The AI can’t have free will like humans, it just follows its programming”, or “lol so you’re scared of Skynet?”, or “Why not just program it to maximize happiness?”.

Having read a lot about AI, these misunderstandings are frustrating to me. This is not that unusual, of course – pretty much any complex topic is going to have people misunderstanding it, and misunderstandings often frustrate me. But there is something unique about the confusions that surround AI, and that’s the extent to which the confusions are philosophical in nature.

Why philosophical? Well, artificial intelligence and philosophy might seem very distinct at first glance, but look closer and you’ll see that they’re connected to one another at a very deep level. Take almost any topic of interest to philosophers – free will, consciousness, epistemology, decision theory, metaethics – and you’ll find an AI researcher looking into the same questions. In fact I would go further and say that those AI researchers are usually doing a better job of approaching the questions. Daniel Dennet said that “AI makes philosophy honest”, and I think there’s a lot of truth to that idea. You can’t write fuzzy, ill-defined concepts into computer code. Thinking in terms of having to program something that actually works takes your head out of the philosophical clouds, and puts you in a mindset of actually answering questions.

All of which is well and good. But the problem with looking at philosophy through the lens of AI is that it’s a two-way street – it means that when you try to introduce someone to the concepts of AI and AI risk, they’re going to be hauling all of their philosophical baggage along with them.

And make no mistake, there’s a lot of baggage. Philosophy is a discipline that’s notorious for many things, but probably first among them is a lack of consensus (I wouldn’t be surprised if there’s not even a consensus among philosophers about how much consensus there is among philosophers). And the result of this lack of consensus has been a kind of grab-bag approach to philosophy among the general public – people see that even the experts are divided, and think that that means they can just choose whatever philosophical position they want.

Want. That’s the key word here. People treat philosophical beliefs not as things that are either true or false, but as choices – things to be selected based on their personal preferences, like picking out a new set of curtains. They say “I prefer to believe in a soul”, or “I don’t like the idea that we’re all just atoms moving around”. And why shouldn’t they say things like that? There’s no one to contradict them, no philosopher out there who can say “actually, we settled this question a while ago and here’s the answer”, because philosophy doesn’t settle things. It’s just not set up to do that. Of course, to be fair people seem to treat a lot of their non-philosophical beliefs as choices as well (which frustrates me to no end) but the problem is particularly pronounced in philosophy. And the result is that people wind up running around with a lot of bad philosophy in their heads.

(Oh, and if that last sentence bothered you, if you’d rather I said something less judgmental like “philosophy I disagree with” or “philosophy I don’t personally happen to hold”, well – the notion that there’s no such thing as bad philosophy is exactly the kind of bad philosophy I’m talking about)

(he said, only 80% seriously)

Anyway, I find this whole situation pretty concerning. Because if you had said to me that in order to convince people of the significance of the AI threat, all we had to do was explain to them some science, I would say: no problem. We can do that. Our society has gotten pretty good at explaining science; so far the Great Didactic Project has been far more successful than it had any right to be. We may not have gotten explaining science down to a science, but we’re at least making progress. I myself have been known to explain scientific concepts to people every now and again, and fancy myself not half-bad at it.

Philosophy, though? Different story. Explaining philosophy is really, really hard. It’s hard enough that when I encounter someone who has philosophical views I consider to be utterly wrong or deeply confused, I usually don’t even bother trying to explain myself – even if it’s someone I otherwise have a great deal of respect for! Instead I just disengage from the conversation. The times I’ve done otherwise, with a few notable exceptions, have only ended in frustration – there’s just too much of a gap to cross in one conversation. And up until now that hasn’t really bothered me. After all, if we’re being honest, most philosophical views that people hold aren’t that important in grand scheme of things. People don’t really use their philosophical views to inform their actions – in fact, probably the main thing that people use philosophy for is to sound impressive at parties.

AI risk, though, has impressed upon me an urgency in regards to philosophy that I’ve never felt before. All of a sudden it’s important that everyone have sensible notions of free will or consciousness; all of a sudden I can’t let people get away with being utterly confused about metaethics.

All of a sudden, in other words, philosophy matters.

I’m not sure what to do about this. I mean, I guess I could just quit complaining, buckle down, and do the hard work of getting better at explaining philosophy. It’s difficult, sure, but it’s not infinitely difficult. I could write blogs posts and talk to people at parties, and see what works and what doesn’t, and maybe gradually start changing a few people’s minds. But this would be a long and difficult process, and in the end I’d probably only be able to affect – what, a few dozen people? A hundred?

And it would be frustrating. Arguments about philosophy are so hard precisely because the questions being debated are foundational. Philosophical beliefs form the bedrock upon which all other beliefs are built; they are the premises from which all arguments start. As such it’s hard enough to even notice that they’re there, let alone begin to question them. And when you do notice them, they often seem too self-evident to be worth stating.

Take math, for example – do you think the number 5 exists, as a number?

Yes? Okay, how about 700? 3 billion? Do you think it’s obvious that numbers just keep existing, even when they get really big?

Well, guess what – some philosophers debate this!

It’s actually surprisingly hard to find an uncontroversial position in philosophy. Pretty much everything is debated. And of course this usually doesn’t matter – you don’t need philosophy to fill out a tax return or drive the kids to school, after all. But when you hold some foundational beliefs that seem self-evident, and you’re in a discussion with someone else who holds different foundational beliefs, which they also think are self-evident, problems start to arise. Philosophical debates usually consist of little more than two people talking past one another, with each wondering how the other could be so stupid as to not understand the sheer obviousness of what they’re saying. And the annoying this is, both participants are correct – in their own framework, their positions probably are obvious. The problem is, we don’t all share the same framework, and in a setting like that frustration is the default, not the exception.

This is not to say that all efforts to discuss philosophy are doomed, of course. People do sometimes have productive philosophical discussions, and the odd person even manages to change their mind, occasionally. But to do this takes a lot of effort. And when I say a lot of effort, I mean a lot of effort. To make progress philosophically you have to be willing to adopt a kind of extreme epistemic humility, where your intuitions count for very little. In fact, far from treating your intuitions as unquestionable givens, as most people do, you need to be treating them as things to be carefully examined and scrutinized with acute skepticism and even wariness. Your reaction to someone having a differing intuition from you should not be “I’m right and they’re wrong”, but rather “Huh, where does my intuition come from? Is it just a featureless feeling or can I break it down further and explain it to other people? Does it accord with my other intuitions? Why does person X have a different intuition, anyway?” And most importantly, you should be asking “Do I endorse or reject this intuition?”. In fact, you could probably say that the whole history of philosophy has been little more than an attempt by people to attain reflective equilibrium among their different intuitions – which of course can’t happen without the willingness to discard certain intuitions along the way when they conflict with others.

I guess what I’m trying to say is: when you’re discussing philosophy with someone and you have a disagreement, your foremost goal should be to try to find out exactly where your intuitions differ. And once you identify that, from there the immediate next step should be to zoom in on your intuitions – to figure out the source and content of the intuition as much as possible. Intuitions aren’t blank structureless feelings, as much as it might seem like they are. With enough introspection intuitions can be explicated and elucidated upon, and described in some detail. They can even be passed on to other people, assuming at least some kind of basic common epistemological framework, which I do think all humans share (yes, even objective-reality-denying postmodernists).

Anyway, this whole concept of zooming in on intuitions seems like an important one to me, and one that hasn’t been emphasized enough in the intellectual circles I travel in. When someone doesn’t agree with some basic foundational belief that you have, you can’t just throw up your hands in despair – you have to persevere and figure out why they don’t agree. And this takes effort, which most people aren’t willing to expend when they already see their debate opponent as someone who’s being willfully stupid anyway. But – needless to say – no one thinks of their positions as being a result of willful stupidity. Pretty much everyone holds beliefs that seem obvious within the framework of their own worldview. So if you want to change someone’s mind with respect to some philosophical question or another, you’re going to have to dig deep and engage with their worldview. And this is a difficult thing to do.

Hence, the philosophical quagmire that we find our society to be in.

It strikes me that improving our ability to explain and discuss philosophy amongst one another should be of paramount importance to most intellectually serious people. This applies to AI risk, of course, but also to many everyday topics that we all discuss: feminism, geopolitics, environmentalism, what have you – pretty much everything we talk about grounds out to philosophy eventually, if you go deep enough or meta enough. And to the extent that we can’t discuss philosophy productively right now, we can’t make progress on many of these important issues.

I think philosophers should – to some extent – be ashamed of the state of their field right now. When you compare philosophy to science it’s clear that science has made great strides in explaining the contents of its findings to the general public, whereas philosophy has not. Philosophers seem to treat their field as being almost inconsequential, as if whatever they conclude at some level won’t matter. But this clearly isn’t true – we need vastly improved discussion norms when it comes to philosophy, and we need far greater effort on the part of philosophers when it comes to explaining philosophy, and we need these things right now. Regardless of what you think about AI, the 21st century will clearly be fraught with difficult philosophical problems – from genetic engineering to the ethical treatment of animals to the problem of what to do with global poverty, it’s obvious that we will soon need philosophical answers, not just philosophical questions. Improvements in technology mean improvements in capability, and that means that things which were once merely thought experiments will be lifted into the realm of real experiments.

I think the problem that humanity faces in the 21st century is an unprecedented one. We’re faced with the task of actually solving philosophy, not just doing philosophy. And if I’m right about AI, then we have exactly one try to get it right. If we don’t, well..

Well, then the fate of humanity may literally hang in the balance.

June 2nd, 42 After Fall
Somewhere in the Colorado Mountains

They first caught sight of the man walking a few miles from the compound. At least it looked like a man. Faded jeans, white t-shirt, light jacket, rucksack. White skin, light brown hair. No obvious disabilities. No logos.

They kept him under surveillance as he approached. In other times they might have shot him on sight, but not now. They were painfully aware of the bounds of sustainable genetic diversity, so instead they drove over in a battered van, rifles loaded, industrial earmuffs in place. Once he was on his knees, they sent Javid the Unhearing over to bind and gag him, then bundled him into the van. No reason to risk exposure.

Javid had not always been deaf, but it was an honor. Some must sacrifice for the good of the others, and he was proud to defend the Sanctum at Rogers Ford.

Once back at the complex, they moved the man to a sound-proofed holding room and unbound him. An ancient PC sat on the desk, marked “Imp Association”. The people did not know who the Imp Association were, but they were grateful for it. Perhaps it was a gift from Olson. Praise be to Olson.

With little else to do, the man sat down and read the instructions on the screen. A series of words showed, and he was commanded to select left or right based on various different criteria. It was very confusing.

In a different room, watchers huddled around a tiny screen, looking at a series of numbers.

REP/DEM 0.0012 0.39 0.003

Good. That was a very good start.

FEM/MRA -0.0082 0.28 -0.029

SJW/NRX 0.0065 0.54 0.012

…

Eventually they passed the lines the catechism denoted “purge with fire and never speak thereof”, on to those merely marked as “highly dangerous”.

KO/PEP 0.1781 0.6 0.297

Not as good, but still within the proscribed tolerances. They would run the supplemental.

T_JCB/T_EWD -0.0008 1.2 -0.001

…

The test continued for some time, until eventually the cleric intoned, “The Trial by Fish is complete. He has passed the Snedecor Fish.” The people nodded as if they understood, then proceeded to the next stage.

This was more dangerous. This required a sacrifice.

She was young – just 15 years old. Fresh faced with long blond hair tied back, Sophia had a cute smile: she was perfect for the duty. Her family were told it was an honor to have their daughter selected.

Sophia entered the room, trepidation in her head, a smile on her face. Casually, she offered him a drink, “Hey, sorry you have to go through all this testin’. You must be hot! Would you like a co cuh?” Her relaxed intonation disguised the fact that these words were the proscribed words, passed down through generations, memorized and cherished as a ward against evil. He accepted the bottle of dark liquid and drank, before tossing the recyclable container in the bin.

In the other room, a box marked ‘ECO’ was ticked off.

“Oh, I’m sorry! I made a mistake – that’s pep-see. I’m so sorry!” she gushed in apology. He assured her it was fine.

In the other room, the cleric satisfied himself that the loyalty brand was burning at zero.

She moved on to the next proscribed question, with the ordained level of casualness, “Say, I know this is a silly question, but do you ever get a song stuck in your head?”

“Errr, what?”

“You know, like you just can’t stop singing it to yourself? Yeah?” Of course, she had no idea what this was like. She was alive.

“Ummm, sorry, no.”

She turned and left the room, relief filling her eyes.

…

After three more days of testing, the man was allowed into the compound. Despite the ravages of an evolution with a generational frequency a hundred times that of humanity, he had somehow preserved himself. He was clean of viral memetic payload. He was alive.

------

Cross-posted on my blog

We looked at the cloudy night sky and thought it would be interesting to share the ways in which, in the past, we made mistakes we would have been able to overcome, if only we had been stronger as rationalists. The experience felt valuable and humbling. So why not do some more of it on Lesswrong?

An antithesis to the Bragging Thread, this is a thread to share where we made mistakes. Where we knew we could, but didn't. Where we felt we were wrong, but carried on anyway.

As with the recent group bragging thread, anything you've done wrong since the comet killed the dinosaurs is fair game, and if it happens to be a systematic mistake that over long periods of time systematically curtailed your potential, that others can try to learn avoiding, better. 

This thread is an attempt to see if there are exceptions to the cached thought that life experience cannot be learned but has to be lived. Let's test this belief together!

Analysis of 2013-2014 LessWrong survey results on how much more self-identified EAers donate

http://www.gwern.net/EA%20donations

Vitalik Buterin has a new post about an interesting theoretical attack against Bitcoin. The idea relies on the assumption that the attacker can credibly commit to something quite crazy. The crazy thing is this: paying out 25.01 BTC to all the people who help him in his attack to steal 25 BTC from everyone, but only if the attack fails. This leads to a weird payoff matrix where the dominant strategy is to help him in the attack. The attack succeeds, and no payoff is made.

Of course, smart contracts make such crazy commitments perfectly possible, so this is a bit less theoretical than it sounds. But even as an abstract though experiment about decision theories, it looks pretty interesting.

By the way, Vitalik Buterin is really on a roll. Just a week ago he had a thought-provoking blog post about how Decentralized Autonomous Organizations could possibly utilize a concept often discussed here: decision theory in a setup where agents can inspect each others' source code. It was shared on LW Discussion, but earned less exposure than I think it deserved.

EDIT 1: One smart commenter of the original post spotted that an isomorphic, extremely cool game was already proposed by billionaire Warren Buffett. Does this thing already have a name in game theory maybe?

EDIT 2: I wrote the game up in detail for some old-school game theorist friends:

The attacker orchestrates a game with 99 players. The attacker himself does not participate in the game.

Rules:

Each of the players can either defect or cooperate, in the usual game theoretic setup where they do announce their decisions simultaneously, without side channels. We call "aggregate outcome" the decision that was made by the majority of the players. If the aggregate outcome is defection, we say that the attack succeeds. A player's payoff consists of two components:

1. If her decision coincides with the aggregate outcome, the player gets 10 utilons.

and simultaneously:

2. if the attack succeeds, the attacker gets 1 utilons from each of the 99 players, regardless of their own decision.

The attacker's commitment is quite counterintuitive: he promises that he will pay 11 utilons to each of the defecting players, but only if the attack fails.

Now the payoff looks like this:

Defection became a dominant strategy. The clever thing, of course, is that if everyone defects, then the attacker reaches his goal without paying out anything.

Quick summary: "Hidden rationalists" are what I call authors who espouse rationalist principles, and probably think of themselves as rational people, but don't always write on "traditional" Less Wrong-ish topics and probably haven't heard of Less Wrong.

I've noticed that a lot of my rationalist friends seem to read the same ten blogs, and while it's great to have a core set of favorite authors, it's also nice to stretch out a bit and see how everyday rationalists are doing cool stuff in their own fields of expertise. I've found many people who push my rationalist buttons in fields of interest to me (journalism, fitness, etc.), and I'm sure other LWers have their own people in their own fields.

So I'm setting up this post as a place to link to/summarize the work of your favorite hidden rationalists. Be liberal with your suggestions!

Another way to phrase this: Who are the people/sources who give you the same feelings you get when you read your favorite LW posts, but who many of us probably haven't heard of?

Here's my list, to kick things off:

• Peter Sandman, professional risk communication consultant. Often writes alongside Jody Lanard. Specialties: Effective communication, dealing with irrational people in a kind and efficient way, carefully weighing risks and benefits. My favorite recent post of his deals with empathy for Ebola victims and is a major, Slate Star Codex-esque tour de force. His "guestbook comments" page is better than his collection of web articles, but both are quite good.
• Doug McGuff, MD, fitness guru and author of the exercise book with the highest citation-to-page ratio of any I've seen. His big thing is "superslow training", where you perform short and extremely intense workouts (video here). I've been moving in this direction for about 18 months now, and I've been able to cut my workout time approximately in half without losing strength. May not work for everyone, but reminds me of Leverage Research's sleep experiments; if it happens to work for you, you gain a heck of a lot of time. I also love the way he emphasizes the utility of strength training for all ages/genders -- very different from what you'd see on a lot of weightlifting sites.
• Philosophers' Mail. A website maintained by applied philosophers at the School of Life, which reminds me of a hippy-dippy European version of CFAR (in a good way). Not much science, but a lot of clever musings on the ways that philosophy can help us live, and some excellent summaries of philosophers who are hard to read in the original. (Their piece on Vermeer is a personal favorite, as is this essay on Simon Cowell.) This recently stopped posting new material, but the School of Life now collects similar work through The Book of Life. 

GiveWell have just announced their latest charity recommendations! What are everyone’s thoughts on them?

A summary: all of the old charities (GiveDirectly, SCI and Deworm the World) remain on the list. They're rejoined by AMF, as the room for more funding issues that led to it being delisted have been resolved to GiveWell's satisfaction. Together these organisations form GiveWell's list of 'top charities', which is now joined by a list of other charities which they see as excellent but not quite in the top tier. The charities on this list are Development Media International, Living Goods, and two salt fortification programs (run by GAIN and ICCIDD).

As normal, GiveWell's site contains extremely detailed writeups on these organisations. Here are some shorter descriptions which I wrote for Charity Science's donations page and my tool for donating tax-efficiently, starting with the new entries:

GiveWell's newly-added charities

Boost health and cognitive development with salt fortification

The charities GAIN and ICCIDD run programs that fortify the salt that millions of poor people eat with iodine. There is strong evidence that this boosts their health and cognitive development; iodine deficiency causes pervasive mental impairment, as well as stillbirth and congenital abnormalities such as severe retardation. It can be done very cheaply on a mass scale, so is highly cost-effective. GAIN is registered in the US and ICCIDD in Canada (although Canadians can give to either via Charity Science, which for complex reasons helps others who donate tax-deductibly to other charities), allowing for especially efficient donations from these countries, and taxpayers from other countries can also often give to them tax-deductibly. For more information, read GiveWell's detailed reviews of GAIN and ICCIDD.

Educate millions in life-saving practices with Development Media International

Development Media International (DMI) produces radio and television broadcasts in developing countries that tell people about improved health practices that can save lives, especially those of young children. Examples of such practices include exclusive breastfeeding. DMI are conducting a randomized controlled trial of their program which has found promising indications of a large decrease in children's deaths. With more funds they would be able to reach millions of people, due to the unparalleled reach of broadcasting. For more information, read GiveWell's detailed review.

Bring badly-needed goods and health services to the poor with Living Goods

Living Goods is a non-profit which runs a network of people selling badly-needed health and household goods door-to-door in their communities in Uganda and Kenya and provide free health advice. A randomized controlled trial suggested that this caused a 25% reduction in under-5 mortality among other benefits. Products sold range from fortified foods and mosquito nets to cookstoves and contraceptives. Giving to Living Goods is an exciting opportunity to bring these badly needed goods and services to some of the poorest families in the world. For more information, read GiveWell's detailed review.

GiveWell's old and returning charities

Treat hundreds of people for parasitic worms

Deworm the World and the Schistosomiasis Control Initiative (SCI) treat parasitic worm infections such as schistosomiasis, which can cause urinary infections, anemia, and other nutritional problems. For more information, read GiveWell's detailed review, or the more accessible Charity Science summary. Deworm the World is registered in the USA and SCI in the UK, allowing for tax-efficient direct donations in those countries, and taxpayers from other countries can also often give to them efficiently.

Make unconditional cash transfers with GiveDirectly

GiveDirectly lets you empower people to purchase whatever they believe will help them most. Eleven randomized controlled trials have supported cash transfers’ impact, and there is strong evidence that recipients know their own situation best and generally invest in things which make them happier in the long term. For more information, read GiveWell's detailed review, or the more accessible Charity Science summary.

Save lives and prevent infections with the Against Malaria Foundation

Malaria causes about a million deaths and two hundred million infections a year. Thankfully a $6 bednet can stop mosquitos from infecting children while they sleep, preventing this deadly disease. This intervention has exceptionally robust evidence behind it, with many randomized controlled trials suggesting that it is one of the most cost-effective ways to save lives. The Against Malaria Foundation (AMF) is an exceptional charity in every respect, and was GiveWell's top recommendation in 2012 and 2013. Not all bednet charities are created equal, and AMF outperforms the rest on every count. They can distribute nets cheaper than most others, for just $6.13 US. They distribute long-lasting nets which don’t need retreating with insecticide. They are extremely transparent and monitor their own impact carefully, requiring photo verification from each net distribution. For more information, read GiveWell's detailed review, or the more accessible Charity Science summary.

How to donate

To find out which charities are tax-deductible in your country and get links to give to them tax-efficiently, you can use this interactive tool that I made. If you give this season, consider sharing the charities you choose on the EA Donation Registry. We can see which charities EAs pick, and which of the new ones prove popular!

Abstract: Sophisticated autonomous AI may need to base its behavior on fuzzy concepts that cannot be rigorously defined, such as well-being or rights. Obtaining desired AI behavior requires a way to accurately specify these concepts. We review some evidence suggesting that the human brain generates its concepts using a relatively limited set of rules and mechanisms. This suggests that it might be feasible to build AI systems that use similar criteria and mechanisms for generating their own concepts, and could thus learn similar concepts as humans do. We discuss this possibility, and also consider possible complications arising from the embodied nature of human thought, possible evolutionary vestiges in cognition, the social nature of concepts, and the need to compare conceptual representations between humans and AI systems.

I just got word that this paper was accepted for the AAAI-15 Workshop on AI and Ethics: I've uploaded a preprint here. I'm hoping that this could help seed a possibly valuable new subfield of FAI research. Thanks to Steve Rayhawk for invaluable assistance while I was writing this paper: it probably wouldn't have gotten done without his feedback motivating me to work on this.

Comments welcome. 

Sorry if the positive illusions are old hat, but I searched and couldn't find any mention of this peer prediction stuff! If nothing else, I think the findings provide a quick heuristic for getting more reliable predictions of your future behavior - just poll a nearby friend!

Peer predictions are often superior to self-predictions. People, when predicting their own future outcomes, tend to give far too much weight to their intentions, goals, plans, desires, etc., and far to little consideration to the way things have turned out for them in the past. As Henry Wadsworth Longfellow observed,

"We judge ourselves by what we feel capable of doing, while others judge us by what we have already done"

...and we are way less accurate for it! A recent study by Helzer and Dunning (2012) took Cornell undergraduates and had them each predict their next exam grade, and then had an anonymous peer predict it too, based solely on their score on the previous exam; despite the fact that the peer had such limited information (while the subjects have presumably perfect information about themselves), the peer predictions, based solely on the subjects' past performance, were much more accurate predictors of subjects' actual exam scores.

In another part of the study, participants were paired-up (remotely, anonymously) and rewarded for accurately predicting each other's scores. Peers were allowed to give just one piece of information to help their partner predict their score; further, they were allowed to request just one piece of information from their partner to aid them in predicting their partner's score. Across the board, participants would give information about their "aspiration level" (their own ideal "target" score) to the peer predicting them, but would be far less likely to ask for that information if they were trying to predict a peer; overwhelmingly, they would ask for information about the participant's past behavior (i.e., their score on the previous exam), finding this information to be more indicative of future performance. The authors note,

There are many reasons to use past behavior as an indicator of future action and achievement. The overarching reason is that past behavior is a product of a number of causal variables that sum up to produce it—and that suite of causal variables in the same proportion is likely to be in play for any future behavior in a similar context.

They go on to say, rather poetically I think, that they have observed "the triumph of hope over experience." People situate their representations of self more in what they strive to be rather than in who they have already been (or indeed, who they are), whereas they represent others more in terms of typical or average behavior (Williams, Gilovich, & Dunning, 2012).

I found a figure I want to include from another interesting article (Kruger & Dunning, 1999); it illustrates this "better than average effect" rather well. Depicted below is an graph summarizing the results of study #3 (perceived grammar ability and test performance as a function of actual test performance):

Along the abscissa, you've got reality: the quartiles represent scores on a test of grammatical ability. The vertical axis, with decile ticks, corresponds to the same peoples' self-predicted ability and test scores. Curiously, while no one is ready to admit mediocrity, neither is anyone readily forecasting perfection; the clear sweet spot is 65-70%. Those in the third quartile seem most accurate in their estimations while those the highest quartile often sold themselves short, underpredicting their actual achievement on average. Notice too that the widest reality/prediction gap is for those the lowest quartile.

It's that time of year again. Actually, a little earlier than that time of year, but I'm pushing it ahead a little to match when Ozy and I expect to have more free time to process the results.

The first draft of the 2014 Less Wrong Census/Survey is complete (see 2013 results here) .

You can see the survey below if you promise not to try to take the survey because it's not done yet and this is just an example!

2014 Less Wrong Census/Survey Draft

I want two things from you.

First, please critique this draft (it's much the same as last year's). Tell me if any questions are unclear, misleading, offensive, confusing, or stupid. Tell me if the survey is so unbearably long that you would never possibly take it. Tell me if anything needs to be rephrased.

Second, I am willing to include any question you want in the Super Extra Bonus Questions section, as long as it is not offensive, super-long-and-involved, or really dumb. Please post any questions you want there. Please be specific - not "Ask something about taxes" but give the exact question you want me to ask as well as all answer choices.

Try not to add more than a few questions per person, unless you're sure yours are really interesting. Please also don't add any questions that aren't very easily sort-able by a computer program like SPSS unless you can commit to sorting the answers yourself.

I will probably post the survey to Main and officially open it for responses sometime early next week.

Summary:

• We're running another Winter Solstice kickstarter - this is to fund the venue, musicians, food, drink and decorations for a big event in NYC on December 20th, as well as to record more music and print a larger run of the Solstice Book of Traditions. 

• There will also be a Rationality Megameetup the following Sunday. (More generally, people are welcome to come to Highgarden and spend the night starting on Friday - although we request that people register their presence via this form so we don't run out of space)

• I'd also like to raise additional money so I can focus full time for the next couple months on helping other communities run their own version of the event, tailored to meet their particular needs while still feeling like part of a cohesive, broader movement - and giving the attendees a genuinely powerful experience. 

The Beginning

Four years ago, twenty NYC rationalists gathered in a room to celebrate the Winter Solstice. We sang songs and told stories about things that seemed very important to us. The precariousness of human life. The thousands of years of labor and curiosity that led us from a dangerous stone age to the modern world. The potential to create something even better, if humanity can get our act together and survive long enough.

One of the most important ideas we honored was the importance of facing truths, even when they are uncomfortable or make us feel silly or are outright terrifying. Over the evening, we gradually extinguished candles, acknowledging harsher and harsher elements of reality.

Until we sat in absolute darkness - aware that humanity is flawed, and alone, in an unforgivingly neutral universe. 

But also aware that we sit beside people who care deeply about truth, and about our future. Aware that across the world, people are working to give humanity a bright tomorrow, and that we have the power to help. Aware that across history, people have looked impossible situations in the face, and through ingenuity and persperation, made the impossible happen.

That seemed worth celebrating. 

The Story So Far

As it turned out, this resonated with people outside the rationality community. When we ran the event again in 2012, non-religious but non-Less Wrong attended the event and told me they found it very moving. In 2013, we pushed it much larger - I ran a kickstarter campaign to fund a big event in NYC. 

A hundred and fifty people from various communities attended. From Less Wrong in particular, we had groups from Boston, San Francisco, North Carolina, Ottawa, and Ohio among other places. The following day was one of the largest East Coast Megameetups. 

Meanwhile, in the Bay Area, several people put together an event that gathered around 80 attendees. In Boston and Vancouever and Leipzig Germany, people ran smaller events. This is shaping up to take root as a legitimate holiday, celebrating human history and our potential future.

This year, we want to do that all again. I also want to dedicate more time to helping other people run their events. Getting people to start celebrating a new holiday is a tricky feat. I've learned a lot about how to go about that and want to help others run polished events that feel connecting and inspirational.

So, what's happening, and how can you help?

• The Big Solstice itself will be Saturday, December 20th at 7:00 PM. To fund it, we're aiming to raise $7500 on kickstarter. This is enough to fund the aforementioned venue, food, drink, live musicians, record new music, and print a larger run of the Solstice Book of Traditions. It'll also pay some expenses for the Megameetup. Please consider contributing to the kickstarter.

• If you'd like to host your own Solstice (either a large or a private one) and would like advice, please contact me at raemon777@gmail.com and we'll work something out.

• There will also be Solstices (of varying sizes) run by Less Wrong / EA folk held in the Bay Area, Seattle, Boston and Leipzig. (There will probably be a larger but non-LW-centered Solstice in Los Angeles and Boston as well).
  • If you'd like to be involved with the Seattle solstice, contact Jai at jai@seattlesecularsolstice.com. If you

• In NYC, there will be a Rationality and EA Megameetup running from Friday, Dec 19th through Sunday evening.
  • Friday night and Saturday morning: Arrival, Settling
  • Saturday at 2PM - 4:30PM: Unconference (20 minute talks, workshops or discussions)
  • Saturday at 7PM: Big Solstice
  • Sunday at Noon: Unconference 2
  • Sunday at 2PM: Strategic New Years Resolution Planning
  • Sunday at 3PM: Discussion of creating private ritual for individual communities
• If you're interested in coming to the Megameetup, please fill out this form saying how many people you're bringing, whether you're interested in giving a talk, and whether you're bringing a vehicle, so we can plan adequately. (We have lots of crash space, but not infinite bedding, so bringing sleeping bags or blankets would be helpful)

Effective Altruism?

Now, at Less Wrong we like to talk about how to spend money effectively, so I should be clear about a few things. I'm raising non-trivial money for this, but this should be coming out of people's Warm Fuzzies Budgets, not their Effective Altruism budgets. This is a big, end of the year community feel-good festival. 

That said, I do think this is an especially important form of Warm Fuzzies. I've had EA-type folk come to me and tell me the Solstice inspired them to work harder, make life changes, or that it gave them an emotional booster charge to keep going even when things were hard. I hope, eventually, to have this measurable in some fashion such that I can point to it and say "yes, this was important, and EA folk should definitely consider it important." 

But I'm not especially betting on that, and there are some failure modes where the Solstice ends up cannibalizing more resources that could have went towards direct impact. So, please consider that this may be especially valuable entertainment, that pushes culture in a direction where EA ideas can go more mainstream and gives hardcore EAs a motivational boost. But I encourage you to support it with dollars that wouldn't have gone towards direct Effective Altruism.

I have taken this well received post by Eliezer, and remade the first third of it into a short and quickly paced youtube video here: http://youtu.be/L2dNANRIALs

The goals of this post are re-introducing the lessons explored in the original (for anyone not yet familiar with them), as well as asking the question of whether this format is actually suited for the lessons LessWrong tries to teach. What are your thoughts?

I just learned of The Future Library project. In short, famous authors will be asked to write new, original fiction that will not be released until 2114. First one announced was Margaret Atwood, of The Handmaiden's Tale fame.

I learned of this when a friend posted on Facebook that "I'm officially looking into being cryogenically frozen due to The Future Library project. See you all in 2114." She meant it as a joke, but after a couple comments she now knows about CI, and she didn't yesterday.

What's one of the most common complaints we hear from Deathists? The future is unknown and scary and there won't be anything there they'd be interested in anyway. Now there will be, if they're Atwood fans.

What's one of the ways artists who give away most of their work (almost all of them nowadays) try to entice people to pay for their albums/books/games/whatever? Including special content that is only available for people who pay (or who pay more). Now there is special content only available for people who are around post-2113.

Which got me to thinking... could we incentivize seeing the future? I know it sounds kinda silly ("What, escaping utter annihilation isn't incentive enough??"), but it seems possible that we could save lives by compiling original work from popular artists (writers, musicians, etc), sealing it tight somewhere, and promising to release it in 100, 200, maybe 250 years. And of course, providing links to cryo resources with all publicity materials.

Would this be worth pursuing? Are there any obvious downsides, aside from cost & difficulty?

In just over two weeks I will be running an online reading group on Nick Bostrom's Superintelligence, on behalf of MIRI. It will be here on LessWrong. This is an advance warning, so you can get a copy and get ready for some stimulating discussion. MIRI's post, appended below, gives the details.

Added: At the bottom of this post is a list of the discussion posts so far.

Nick Bostrom’s eagerly awaited Superintelligence comes out in the US this week. To help you get the most out of it, MIRI is running an online reading group where you can join with others to ask questions, discuss ideas, and probe the arguments more deeply.

The reading group will “meet” on a weekly post on the LessWrong discussion forum. For each ‘meeting’, we will read about half a chapter of Superintelligence, then come together virtually to discuss. I’ll summarize the chapter, and offer a few relevant notes, thoughts, and ideas for further investigation. (My notes will also be used as the source material for the final reading guide for the book.)

Discussion will take place in the comments. I’ll offer some questions, and invite you to bring your own, as well as thoughts, criticisms and suggestions for interesting related material. Your contributions to the reading group might also (with permission) be used in our final reading guide for the book.

We welcome both newcomers and veterans on the topic. Content will aim to be intelligible to a wide audience, and topics will range from novice to expert level. All levels of time commitment are welcome.

We will follow this preliminary reading guide, produced by MIRI, reading one section per week.

If you have already read the book, don’t worry! To the extent you remember what it says, your superior expertise will only be a bonus. To the extent you don’t remember what it says, now is a good time for a review! If you don’t have time to read the book, but still want to participate, you are also welcome to join in. I will provide summaries, and many things will have page numbers, in case you want to skip to the relevant parts.

If this sounds good to you, first grab a copy of Superintelligence. You may also want to sign up here to be emailed when the discussion begins each week. The first virtual meeting (forum post) will go live at 6pm Pacific on Monday, September 15th. Following meetings will start at 6pm every Monday, so if you’d like to coordinate for quick fire discussion with others, put that into your calendar. If you prefer flexibility, come by any time! And remember that if there are any people you would especially enjoy discussing Superintelligence with, link them to this post!

Topics for the first week will include impressive displays of artificial intelligence, why computers play board games so well, and what a reasonable person should infer from the agricultural and industrial revolutions.

Posts in this sequence

Week 1: Past developments and present capabilities

Week 2: Forecasting AI

Week 3: AI and uploads

Week 4: Biological cognition, BCIs, organizations

Week 5: Forms of superintelligence

Week 6: Intelligence explosion kinetics

Week 7: Decisive strategic advantage

Week 8: Cognitive superpowers

Week 9: The orthogonality of intelligence and goals

Week 10: Instrumentally convergent goals

Week 11: The treacherous turn

Week 12: Malignant failure modes

Week 13: Capability control methods

Week 14: Motivation selection methods

Week 15: Oracles, genies and sovereigns

Week 16: Tool AIs

Week 17: Multipolar scenarios

Week 18: Life in an algorithmic economy

Week 19: Post-transition formation of a singleton

Week 20: The value-loading problem

Week 21: Value learning

Week 22: Emulation modulation and institution design

Week 23: Coherent extrapolated volition

Week 24: Morality models and "do what I mean"

Week 25: Components list for acquiring values

Week 26: Science and technology strategy

Week 27: Pathways and enablers

Week 28: Collaboration

Week 29: Crunch time

The official story: "Fifty Shades of Grey" was a Twilight fan-fiction that had over two million downloads online. The publishing giant Vintage Press saw that number and realized there was a huge, previously-unrealized demand for stories like this. They filed off the Twilight serial numbers, put it in print, marketed it like hell, and now it's sold 60 million copies.

The reality is quite different.

Scott suggests that ranking morality is similar to ranking web pages. A quote:

Philosophers from Socrates on, I was vaguely aware, had struggled to define what makes a person “moral” or “virtuous,” without tacitly presupposing the answer.  Well, it seemed to me that, as a first attempt, one could do a lot worse than the following:

A moral person is someone who cooperates with other moral people, and who refuses to cooperate with immoral people.

Proposed solution:

Just like in CLEVER or PageRank, we can begin by giving everyone in the community an equal number of “morality starting credits.”  Then we can apply an iterative update rule, where each person A can gain morality credits by cooperating with each other person B, and A gains more credits the more credits B has already.  We apply the rule over and over, until the number of morality credits per person converges to an equilibrium.  (Or, of course, we can shortcut the process by simply finding the principal eigenvector of the “cooperation matrix,” using whatever algorithm we like.)  We then have our objective measure of morality for each individual, solving a 2400-year-old open problem in philosophy.

He then talks about "eigenmoses and eigenjesus" and other fun ideas, like Plato at the Googleplex.

One final quote:

All that's needed to unravel the circularity is a principal eigenvector computation on the matrix of trust.

EDIT: I am guessing that after judicious application of this algorithm one would end up with the other Scott A's loosely connected components with varying definitions of morality, the Archipelago. UPDATE: He chimes in.

EDIT2: The obvious issue of equating prevailing mores with morality is discussed to death in the comments. Please read them first before raising it yet again here.

Any scenario where advanced AI takes over the world requires some mechanism for an AI to leverage its position as ethereal resident of a computer somewhere into command over a lot of physical resources.

One classic story of how this could happen, from Eliezer:

1. Crack the protein folding problem, to the extent of being able to generate DNA strings whose folded peptide sequences fill specific functional roles in a complex chemical interaction.
2. Email sets of DNA strings to one or more online laboratories which offer DNA synthesis, peptide sequencing, and FedEx delivery. (Many labs currently offer this service, and some boast of 72-hour turnaround times.)
3. Find at least one human connected to the Internet who can be paid, blackmailed, or fooled by the right background story, into receiving FedExed vials and mixing them in a specified environment.
4. The synthesized proteins form a very primitive “wet” nanosystem which, ribosomelike, is capable of accepting external instructions; perhaps patterned acoustic vibrations delivered by a speaker attached to the beaker.
5. Use the extremely primitive nanosystem to build more sophisticated systems, which construct still more sophisticated systems, bootstrapping to molecular nanotechnology—or beyond.

You can do a lot of reasoning about AI takeover without any particular picture of how the world gets taken over. Nonetheless it would be nice to have an understanding of these possible routes. For preparation purposes, and also because a concrete, plausible pictures of doom are probably more motivating grounds for concern than abstract arguments.

So MIRI is interested in making a better list of possible concrete routes to AI taking over the world. And for this, we ask your assistance.

What are some other concrete AI takeover mechanisms? If an AI did not have a solution to the protein folding problem, and a DNA synthesis lab to write off to, what else might it do? 

We would like suggestions that take an AI from being on an internet-connected computer to controlling substantial physical resources, or having substantial manufacturing ability.

We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years. So limited involvement of advanced nanotechnology and quantum computers would be appreciated. 

We welcome partial suggestions, e.g. 'you can take control of a self-driving car from the internet - probably that could be useful in some schemes'. 

Thank you!

As in, how do you find ways to meet the right people you talk to? Presumably, they would have personality fit with you, and be high on both intelligence and openness. Furthermore, they would be in the point of their life where they are willing to spend time with you (although sometimes you can learn a lot from people simply by friending them on Facebook and just observing their feeds from time to time).

Historically, I've made myself extremely stalkable on the Internet. In retrospect, I believe that this "decision" is on the order of one of the very best decisions I've ever made in my life, and has made me better at social discovery than most people I know, despite my dual social anxiety and Asperger's. In fact, if a more extroverted non-Aspie could do the same thing, I think they could do WONDERS with developing an online profile.

I've also realized more that social discovery is often more rewarding when done with teenagers. You can do so much to impact teenagers, and they often tend to be a lot more open to your ideas/musings (just as long as you're responsible).

But I've wondered - how else have you done it? Especially in real life? What are some other questions you ask with respect to social discovery? I tend to avoid real life for social discovery simply because it's extremely hit-and-miss, but I've discovered (from Richard Florida's books) that the Internet often strengthens real-life interaction because it makes it so much easier to discover other people in real life (and then it's in real life when you can really get to know people).

http://www.huffingtonpost.com/stephen-hawking/artificial-intelligence_b_5174265.html

Very surprised none has linked to this yet:

TL;DR: AI is a very underfunded existential risk.

Nothing new here, but it's the biggest endorsement the cause has gotten so far. I'm greatly pleased they got Stuart Russell, though not Peter Norvig, who seems to remain lukewarm to the cause. Also too bad this was Huffington vs something more respectable. With some thought I think we could've gotten the list to be more inclusive and found a better publication; still I think this is pretty huge.

ErinFlight said:

I'm struggling to understand anything technical on this website. I've enjoyed reading the sequences, and they have given me a lot to thing about. Still, I've read the introduction to Bayes theorem multiple times, and I simply can't grasp it. Even starting at the very beginning of the sequences I quickly get lost because there are references to programming and cognitive science which I simply do not understand.

Thinking about it, I realized that this might be a common concern. There are probably plenty of people who've looked at various more-or-less technical or jargony Less Wrong posts, tried understanding them, and then given up (without posting a comment explaining their confusion).

So I figured that it might be good to have a thread where you can ask for explanations for any Less Wrong post that you didn't understand and would like to, but don't want to directly comment on for any reason (e.g. because you're feeling embarassed, because the post is too old to attract much traffic, etc.). In the spirit of various Stupid Questions threads, you're explicitly encouraged to ask even for the kinds of explanations that you feel you "should" get even yourself, or where you feel like you could get it if you just put in the effort (but then never did).

You can ask to have some specific confusing term or analogy explained, or to get the main content of a post briefly summarized in plain English and without jargon, or anything else. (Of course, there are some posts that simply cannot be explained in non-technical terms, such as the ones in the Quantum Mechanics sequence.) And of course, you're encouraged to provide explanations to others!

Ever since Tversky and Kahneman started to gather evidence purporting to show that humans suffer from a large number of cognitive biases, other psychologists and philosophers have criticized these findings. For instance, philosopher L. J. Cohen argued in the 80's that there was something conceptually incoherent with the notion that most adults are irrational (with respect to a certain problem). By some sort of Wittgensteinian logic, he thought that the majority's way of reasoning is by definition right. (Not a high point in the history of analytic philosophy, in my view.) See chapter 8 of this book (where Gigerenzer, below, is also discussed).

Another attempt to resurrect human rationality is due to Gerd Gigerenzer and other psychologists. They have a) shown that if you tweak some of the heuristics and biases (i.e. the research program led by Tversky and Kahneman) experiments but a little - for instance by expressing probabilities in terms of frequencies - people make much fewer mistakes and b) argued, on the back of this, that the heuristics we use are in many situations good (and fast and frugal) rules of thumb (which explains why they are evolutionary adaptive). Regarding this, I don't think that Tversky and Kahneman ever doubted that the heuristics we use are quite useful in many situations. Their point was rather that there are lots of naturally occuring set-ups which fool our fast and frugal heuristics. Gigerenzer's findings are not completely uninteresting - it seems to me he does nuance the thesis of massive irrationality a bit - but his claims to the effect that these heuristics are rational in a strong sense are wildly overblown in my opnion. The Gigerenzer vs. Tversky/Kahneman debates are well discussed in this article (although I think they're too kind to Gigerenzer).

A strong argument against attempts to save human rationality is the argument from individual differences, championed by Keith Stanovich. He argues that the fact that some intelligent subjects consistently avoid to fall prey to the Wason Selection task, the conjunction fallacy, and other fallacies, indicates that there is something misguided with the notion that the answer that psychologists traditionally has seen as normatively correct is in fact misguided.

Hence I side with Tversky and Kahneman in this debate. Let me just mention one interesting and possible succesful method for disputing some supposed biases. This method is to argue that people have other kinds of evidence than the standard interpretation assumes, and that given this new interpretation of the evidence, the supposed bias in question is in fact not a bias. For instance, it has been suggested that the "false consensus effect" can be re-interpreted in this way:

The False Consensus Effect

Bias description: People tend to imagine that everyone responds the way they do. They tend to see their own behavior as typical. The tendency to exaggerate how common one’s opinions and behavior are is called the false consensus effect. For example, in one study, subjects were asked to walk around on campus for 30 minutes, wearing a sign board that said "Repent!". Those who agreed to wear the sign estimated that on average 63.5% of their fellow students would also agree, while those who disagreed estimated 23.3% on average.

Counterclaim (Dawes & Mulford, 1996): The correctness of reasoning is not estimated on the basis of whether or not one arrives at the correct result. Instead, we look at whether reach reasonable conclusions given the data they have. Suppose we ask people to estimate whether an urn contains more blue balls or red balls, after allowing them to draw one ball. If one person first draws a red ball, and another person draws a blue ball, then we should expect them to give different estimates. In the absence of other data, you should treat your own preferences as evidence for the preferences of others. Although the actual mean for people willing to carry a sign saying "Repent!" probably lies somewhere in between of the estimates given, these estimates are quite close to the one-third and two-thirds estimates that would arise from a Bayesian analysis with a uniform prior distribution of belief. A study by the authors suggested that people do actually give their own opinion roughly the right amount of weight.

(The quote is from an excellent Less Wrong article on this topic due to Kaj Sotala. See also this post by him, this by Andy McKenzie, this by Stuart Armstrong and this by lukeprog on this topic. I'm sure there are more that I've missed.)

It strikes me that the notion that people are "massively flawed" is something of an intellectual cornerstone of the Less Wrong community (e.g. note the names "Less Wrong" and "Overcoming Bias"). In the light of this it would be interesting to hear what people have to say about the rationality wars. Do you all agree that people are massively flawed?

Let me make two final notes to keep in mind when discussing these issues. Firstly, even though the heuristics and biases program is sometimes seen as pessimistic, one could turn the tables around: if they're right, we should be able to improve massively (even though Kahneman himself seems to think that that's hard to do in practice). I take it that CFAR and lots of LessWrongers who attempt to "refine their rationality" assume that this is the case. On the other hand, if Gigerenzer or Cohen are right, and we already are very rational, then it would seem that it is hard to do much better. So in a sense the latter are more pessimistic (and conservative) than the former.

Secondly, note that parts of the rationality wars seem to be merely verbal and revolve around how "rationality" is to be defined (tabooing this word is very often a good idea). The real question is not if the fast and frugal heuristics are in some sense rational, but whether there are other mental algorithms which are more reliable and effective, and whether it is plausible to assume that we could learn to use them on a large scale instead.

In a discussion a couple months ago, Luke said, "I think it's hard to tell whether donations do more good at MIRI, FHI, CEA, or CFAR." So I want to have a thread to discuss that.

My own very rudimentary thoughts: I think the research MIRI does is probably valuable, but I don't think it's likely to lead to MIRI itself building FAI. I'm convinced AGI is much more likely to be built by a government or major corporation, which makes me more inclined to think movement-building activities are likely to be valuable, to increase the odds of the people at that government or corporation being conscious of AI safety issues, which MIRI isn't doing.

It seems like FHI is the obvious organization to donate to for that purpose, but Luke seems to think CEA (the Centre for Effective Altruism) and CFAR could also be good for that, and I'm not entirely clear on why. I sometimes get the impression that some of CFAR's work ends up being covert movement-building for AI-risk issues, but I'm not sure to what extent that's true. I know very little about CEA, and a brief check of their website leaves me a little unclear on why Luke recommends them, aside from the fact that they apparently work closely with FHI.

This has some immediate real-world relevance to me: I'm currently in the middle of a coding bootcamp and not making any money, but today my mom offered to make a donation to a charity of my choice for Christmas. So any input on what to tell her would be greatly appreciated, as would more information on CFAR and CEA, which I'm sorely lacking in.

FHI has released a new tech report:

Armstrong, Bostrom, and Shulman. Racing to the Precipice: a Model of Artificial Intelligence Development.

Abstract:

This paper presents a simple model of an AI arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivized to finish first — by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases.

The paper is short and readable; discuss it here!

But my main reason for posting is to ask this question: What is the most similar work that you know of? I'd expect people to do this kind of thing for modeling nuclear security risks, and maybe other things, but I don't happen to know of other analyses like this.

The soon-to-be-resigning Dominic Cummings, advisor to the Education Secretary of the Coalition government, has released a 250-page manifesto describing the problems of the British educational establishment ("the blob" in Whitehall parlance) and offering solutions. I post this here because both his analysis and recommendations are likely to be interesting to LW, in particular an increased emphasis on STEM, broader knowledge of the limits of human reasoning and how they relate to managing complex systems, an appreciation for "agenty"-ness in organizational leadership, whole-brain emulation, intelligence enhancement, recursive self-improving AGI, analysis of human interactions on a firm evolutionary-psychological basis, and a rejection of fashionable pseudoscientific theories of psychology and society. Relevant extracts:

This essay is aimed mainly at ~15-25 year-olds and those interested in more ambitious education and training for them. Not only are most of them forced into mediocre education but they are also then forced into dysfunctional institutions where many face awful choices: either conform to the patterns set by middle-aged mediocrities (don’t pursue excellence, don’t challenge bosses’ errors, and so on) or soon be despised and unemployed. Some of the ideas sketched here may help some to shape their own education and create their own institutions. As people such as Linus Torvald (Linux) or Mark Zuckerberg (Facebook) have shown, the young are capable of much more than the powerful and middle-aged, who control so many institutions, like to admit.2 A significant change in education and training could also help us partly ameliorate the grim cycle of predictable political dysfunction. 

Although it is generally psychologically unpleasant to focus on our problems and admit the great weaknesses of our institutions, including what we in politics and our leaders do not understand, it is only through honest analysis that progress is possible. As Maxwell said, ‘Thoroughly conscious ignorance is a prelude to every real advance in knowledge.’ Reliable knowledge about what works (how do people learn, what teaching methods work, how to use technology) should be built cumulatively on the foundation of empirical tests as suggested by physicist Carl Wieman and others (cf. Section 6 and Endnote). New systems and curricula would work in different ways for a school, a University, or an equivalent to a boxing gym for politicians if such a thing existed.

In particular, it is suggested that we need an ‘Odyssean’ education so that a substantial fraction of teenagers, students and adults might understand something of our biggest intellectual and practical problems, and be trained to take effective action.

The Nobel-winning physicist, Murray Gell Mann, one of the architects of the Standard Model of particle physics and namer of the ‘quark’,3 has described a scientific and political need for an ‘Odyssean’ philosophy that can synthesise a) maths and the natural sciences, b) the social sciences, and c) the humanities and arts, into necessarily crude, trans-disciplinary, integrative thinking about complex systems.

‘Today the network of relationships linking the human race to itself and to the rest of the biosphere is so complex that all aspects affect all others to an extraordinary degree. Someone should be studying the whole system, however crudely that has to be done, because no gluing together of partial studies of a complex nonlinear system can give a good idea of the behavior of the whole...

‘Those who study complex adaptive systems are beginning to find some general principles that underlie all such systems, and seeking out those principles requires intensive discussions and collaborations among specialists in a great many fields. Of course the careful and inspired study of each specialty remains as vital as ever. But integration of those specialities is urgently needed as well. Important contributions are made by the handful of scholars and scientists who are transforming themselves from specialists into students of simplicity and complexity or of complex adaptive systems in general... 

‘[There is] the distinction (made famous by Nietzsche) between “Apollonians”, who favor logic, the analytical approach, and a dispassionate weighing of the evidence, and “Dionysians”’, who lean more toward intuition, synthesis, and passion…[4] But some of us seem to belong to another category: the “Odysseans”, who combine the two predilections in their quest for connections among ideas… We need to celebrate the contribution of those who dare take what I call “a crude look at the whole”… 

‘… broadly integrative thinking is relegated to cocktail parties. In academic life, in bureaucracies, and elsewhere, the task of integration is insufficiently respected. Yet anyone at the top of an organization … has to make decisions as if all aspects of a situation, along with the interaction among those aspects, were being taken into account. Is it reasonable for the leader, reaching down into the organization for help, to encounter specialists and for integrative thinking to take place only when he or she makes the final intuitive judgements?

‘[A] multiplicity of crude but integrative policy studies, involving not just linear projection but evolution and highly nonlinear simulation and gaming, may provide some modest help in generating a collective foresight function for the human race… 

‘Given the immense complexity of the numerous interlocking issues facing humanity, foresight demands the ability to identify and gather great quantities of relevant information; the ability to catch glimpses, using that information, of the choices offered by the branching alternative histories of the future, and the wisdom to select simplifications and approximations that do not sacrifice the representation of critical qualitative issues, especially issues of values…

‘Computers ... can serve us both by learning or adapting themselves and by modelling or simulating systems in the real world that learn or adapt or evolve. .. Powerful computers are essential for assistance in looking into the future, but we must not allow their use to bias the formulation of problems toward the quantifiable and analyzable at the expense of the important.’5

One of the things that ‘synthesizers’ need to learn is the way that many themes cut across subjects and generate new subjects. Trans-disciplinary studies of complex systems have been profoundly affected by connected intellectual revolutions in physics, maths, logic, computation, and biology, though these connections have so far barely been integrated in school or university curricula. Ideas about ‘information’ cut across physics (thermodynamics and entropy),7 computation (bits, qubits, and ‘artificial agents’), economics (‘economic agents’, Hayek’s description of prices as an ‘information discovery process’), evolution (genetic networks),8 the brain (neuronal networks), ‘intelligence failures’9 and other subjects. Physics is inseparable from old philosophical questions (it is ‘experimental metaphysics’) and it is merging with computer science to produce quantum computation and ‘quantum information theory’.10 Evolutionary ideas took hold in economics (Hume, Smith) and biology (Darwin), and they have now been incorporated into computer science (e.g. ‘genetic algorithms’) in order to ‘evolve’ solutions to problems in large search spaces, and they have suggested ideas for engineering solutions. (For example, it is suggested that dangers such as bioterrorism or pandemics should be defended by developing ‘artificial immune systems’ in which defences operate according to the evolutionary principles of i) generating lots of solutions with random variation, and ii) differential replication of the most effective agents, instead of reliance on traditional centralised institutions that make similar mistakes repeatedly.) Machine intelligence (or ‘automated reasoning’) has been shaped by, and in turn is reshaping, ideas about the mind and is now used to design computers, including those that are used to investigate the mind. Behavioural genetics, evolutionary psychology, cognitive science and neuroscience are reshaping not only economics (e.g. fMRI scans of people playing financial games) and history (e.g. the influence of evolved antipathy for out-groups) but also how we design institutions (e.g. how we evolved to succeed in the sexual politics of small, primitive collectivist tribes hence many of the standard features of internal politics). As will be discussed, there are various developments in education that seek to reflect these trans-disciplinary themes: e.g. the Nobel-winning neuroscientist, Eric Kandel, plans a new PhD programme combining neuroscience, psychology and art history as part of the ‘Mind, Brain, Behaviour Project’ at Columbia.11

Neuroscience, cognitive science, behavioural genetics, and evolutionary biology have developed our understanding of the mind. They and other disciplines have combined with Moore’s Law and scanning technology to provide increasingly accurate maps and quantitative models of the brain107 (which Obama promised federal support for in 2013) and rapidly improving brain-computer interfaces.108 We can look inside our minds with tools that our minds have given us and watch the brain watching itself. These developments have undermined the basis for Descartes’ Ghost in the Machine, Locke’s Blank Slate, and Rousseau’s Noble Savage (pace the current, sometimes misguided, backlash).109

The brain consists of ~80 billion (8x1010) neurons and ~100 trillion (1014) synapses. Operating on ~20 watts it performs ~1017 floating point computations per second. As well as the brain-computer interfaces already underway, various projects plan to map the brain completely (e.g the Connectome Project), simulate the brain (e.g Markram's project), and build new computer architectures modeled on the brain and performing similarly to the brain.

For example, the most successful government technology developer, DARPA, has made robotics and machine intelligence a priority with projects such as the SyNAPSE Project (led by IBM’s Modha) to create ‘a brain inspired electronic “chip” that mimics that function, size, and power consumption of a biological cortex.’110 They have recently announced progress in building this new architecture, fundamentally different to the standard ‘von Neumann architecture’ of all normal computers.111

The Human Brain Project is aiming to model a human brain and in 2012 was awarded €1 billion by the EU. In 2005, a single neuron was simulated; in 2008, a cortical column (104 neurons); in 2011, 100 columns (106 neurons). Markram plans for a full rodent brain simulation in 2014 and a human brain simulation 2020-25.

However, so far SyNAPSE and The Human Brain Project have not demonstrated how simulations connect to observable behaviours. In November 2012, Eliasmith et al (Science, 30/12/2012) published details of a large-scale computational model of the brain (Semantic Pointer Architecture Unified Network, or ‘Spaun’) intended to bridge ‘the brain-behaviour gap’ by simulating complex behaviour. Spaun is a ‘spiking neuron model’ of 2.5m simulated neurons organised into subsystems resembling different brain areas. All inputs are images of characters; all outputs are movements of a physically modeled arm. Incoming visual images are compressed by successive layers of the network that extract increasingly complex information, and simple internally generated commands (e.g. ‘draw a number’) are ‘expanded’ into complex mechanical movements. Spaun simulates the working memory and an ‘action selection system’. It performs eight tasks such as image recognition, copy drawing, reinforcement learning, and a fluid reasoning task ‘isomorphic to the induction problems from the Raven’s Progressive Matrices (RPM) test for fluid intelligence’. Spaun managed to pass some basic aspects of an IQ test. Spaun is not task-specific so the model could be extended to other tasks and scaled-up in other ways.

In 2013, Alex Wissner-Gross, a Harvard computer scientist, published a paper in Physical Review in ‘an attempt to describe intelligence as a fundamentally thermodynamic process’, proposing that intelligence can spontaneously emerge from the attempt to maximise freedom of action in the future. He built a software programme, ‘ENTROPICA’, designed to maximise the production of long-term entropy of any system it finds itself in. ENTROPICA then solved various problems including intelligence tests, playing games, social cooperation, trading financial instruments, and ‘balancing’ a physical system and so on.

‘We were actually able to successfully reproduce standard intelligence tests and other cognitive behaviors, all without assigning any explicit goals… ‘

Think of games like chess or Go in which good players try to preserve as much freedom of action as possible. When the best computer programs play Go, they rely on a principle in which the best move is the one which preserves the greatest fraction of possible wins. When computers are equipped with this simple strategy - along with some pruning for efficiency - they begin to approach the level of Go grandmasters…

‘Our causal entropy maximization theory predicts that AIs may be fundamentally antithetical to being boxed. If intelligence is a phenomenon that spontaneously emerges through causal entropy maximization, then it might mean that you could effectively reframe the entire definition of Artificial General Intelligence to be a physical effect resulting from a process that tries to avoid being boxed...

‘The conventional storyline has been that we would first build a really intelligent machine, and then it would spontaneously decide to take over the world…We may have gotten the order of dependence all wrong. Intelligence and superintelligence may actually emerge from the effort of trying to take control of the world - and specifically, all possible futures - rather than taking control of the world being a behavior that spontaneously emerges from having superhuman machine intelligence… 

‘The recursive self-improving of an AI can be seen as implicitly inducing a flow over the entire space of possible AI programs. In that context, if you look at that flow over AI program space, it is conceivable that causal entropy maximization might represent a fixed point and that a recursively self-improving AI will tend to self-modify so as to do a better and better job of maximizing its future possibilities.

‘In the problem solving example, I show that cooperation can emerge as a means for the systems to maximize their causal entropy, so it doesn't always have to be competition. If more future possibilities are gained through cooperation rather than competition, then cooperation by itself should spontaneously emerge, speaking to the potential for friendliness.’ (Interview.)

The education of the majority even in rich countries is between awful and mediocre. A tiny number, less than 1 percent, are educated in the basics of how the ‘unreasonable effectiveness of mathematics’ provides the ‘language of nature’ and a foundation for our scientific civilisation116 and only a small subset of that <1% then study trans-disciplinary issues concerning the understanding, prediction and control of complex nonlinear systems. Unavoidably, the level of one’s mathematical understanding imposes limits on the depth to which one can explore many subjects. For example, it is impossible to follow academic debates about IQ unless one knows roughly what ‘normal distribution’ and ‘standard deviation’ mean, and many political decisions, concerning issues such as risk, cannot be wisely taken without at least knowing of the existence of mathematical tools such as conditional probability. Only a few aspects of this problem will be mentioned.

There is widespread dishonesty about standards in English schools,117 low aspiration even for the brightest children,118 and a common view that only a small fraction of the population, a subset of the most able, should be given a reasonably advanced mathematical and scientific education, while many other able pupils leave school with little more than basic numeracy and some scattered, soon-forgotten facts. A reasonable overall conclusion from international comparisons, many studies, and how universities have behaved, is that overall standards have roughly stagnated over the past thirty years (at best), there are fewer awful schools, the sharp rises in GCSE results reflect easier exams rather than real educational improvements, and the skills expected of the top 20 percent of the ability range studying core A Level subjects significantly declined (while private schools continued to teach beyond A Levels), hence private schools have continued to dominate Oxbridge entry while even the best universities have had to change degree courses substantially

There is hostility to treating education as a field for objective scientific research to identify what different methods and resources might achieve for different sorts of pupils. The quality of much education research is poor. Randomised control trials (RCTs) are rarely used to evaluate programmes costing huge amounts of money. They were resisted by the medical community for decades (‘don’t challenge my expertise with data’)119 and this attitude still pervades education. There are many ‘studies’ that one cannot rely on and which have not been replicated. Methods are often based on technological constraints of centuries ago, such as lectures. Square wheels are repeatedly reinvented despite the availability of exceptional materials and subject experts are routinely ignored by professional ‘educationalists’.120 There is approximately zero connection between a) debates in Westminster and the media about education and b) relevant science, and little desire to make such connections or build the systems necessary; almost everybody prefers the current approach despite occasional talk of ‘evidence-based policy’ (this problem is one of the reasons we asked Ben Goldacre to review the DfE’s analysis division). The political implications of discussing the effects of evolutionary influences on the variance of various characteristics (such as intelligence (‘g’) and conscientiousness) and the gaps between work done by natural scientists and much ‘social science’ commentary have also prevented rational public discussion (cf. Endnote on IQ).121

Westminster and Whitehall have distorted incentives to learn and improve,122 have simultaneously taken control of curricula and exams and undermined the credibility of both, and have then blamed universities for the failures of state schools123 and put enormous pressure on Universities and academics not to speak publicly about problems with exams, which has made rational discussion of exams impossible. Most people with power in the education system are more worried about being accused of ‘elitism’ (and ‘dividing children into sheep and goats’) than they are about problems caused by poor teaching and exams and they would rather live with those problems than deal with those accusations.124

[124 E.g. Almost everybody the DfE consulted 2011-13 about curriculum and exam reform was much more concerned about accusations of elitism than about the lack of ambition for the top 20%. Although they would not put it like this, most prominent people in the education world tacitly accept that failing to develop the talents of the most able is a price worth paying to be able to pose as defenders of ‘equality’. The insistence that ~95% of pupils be able to take the same exam at 16 means (if one assumes symmetrical exclusions) that the exam must embrace plus and minus two standard deviations on the cognitive ability range: i.e. they exclude only the bottom 2.5% (i.e. an IQ of <~70) and top 2.5% (i.e an IQ of >~130, which is the average Physics PhD).]

There is huge variation in school performance (on exams that are sub-optimal) among schools with the poorest children. About a quarter of primaries have over a quarter of their pupils leave each year who are not properly prepared for basic secondary studies and few such pupils enjoy a turnaround at secondary;125 other primaries (including in the poorest areas) have <5% in such a desperate situation. Consider a basic benchmark: getting four-fifths of pupils to at least a ‘C’ in existing English and Maths GCSE. A small minority of state schools achieve this, while others with similar funding and similarly impoverished pupils struggle to get two-fifths to this level. This wide variety in performance combined with severe limits on direct parent choice means the system is partly rationed by house price.126

This wide variety in performance also strongly suggests that the block to achieving this basic benchmark is the management and quality of teaching in the school; the block is not poverty,127 IQ, money,128 lack of innovation, or a lack of understanding about how to teach basics. Making a transition to a school system in which ~4/5 meet this basic level is therefore an issue of doing things we already know how to do; the obstacles are political and bureaucratic (such as replacing management and bad teachers despite political resistance and legal complexity), although this must not blind us to the fact that most variation in performance is due to within school factors (including genetics) rather than between school factors (see below).

There are various problems with maths and science education…

The Royal Society estimates that ~300,000 per year need some sort of post-GCSE Maths course but only ~100,000 do one now. About 6/10 now get at least a C in English and Maths GCSE; most never do any more maths after GCSE.129 There is no widely respected ‘maths for non-maths specialists’ 16-18 course (see below).130 About 70-80,000 (~1/10 of the cohort) do Maths A Level each year (of these ~⅓ come from private schools and grammars)131 and ~1-2% also do Further Maths. In the last year for which we have data, ~0.5% (3,580 pupils) went on to get A or A* in each of A Level Maths, Further Maths, and Physics.132 Further, many universities only demand GCSE Maths as a condition of entry even for scientific degrees, so ~20% of HE Engineering entrants, ~40% of Chemistry and Economics entrants, and ~60-70% of Biology and Computer Science entrants do not have A Level Maths. Less than10% of undergraduate bioscience degree courses demand A Level Maths. 

Because of how courses have been devised, ~4/5 pupils leave England’s schools without basic knowledge of subjects like logarithms and exponential functions which are fundamental to many theoretical and practical problems (such as compound interest and interpreting a simple chart on a log scale), and unaware of the maths and physics of Newton (basic calculus and mechanics). Less than one in ten has a grasp of the maths of probability developed in the 19th Century such as ‘normal distributions’ and the Central Limit Theorem (‘bell curves’) and conditional probability.133 Only the 1-2% doing Further Maths study complex numbers, matrices and basic linear algebra. Basic logic and set theory (developed c. 1850-1940) do not feature in Maths or Further Maths A levels, so almost nobody leaves school with even a vague idea of the modern axiomatic approach to maths unless they go to a very unusual school or teach themselves.

133 Gigerenzer’s ‘Reckoning With Risk’ has terrifying stats on the inability of trained medical professionals making life and death decisions to understand the basics of conditional probability, which is not covered in the pre-16 curriculum (cf. Endnote). Current A Level modules have conditional probability and normal distributions in S1 and S2 (not compulsory), so one could have an A* in A Level Maths and Further Maths without knowing what these are. Data on who does which modules is not published by exam boards.

(...)

The education world generally resists fiercely the idea that a large fraction of children can or should be introduced to advanced ideas but we could substantially raise expectations without embracing ‘Ender’s Game’. It is almost never asked: how could we explore rigorously how ambitious it is realistic to be? If you ask, including in the Royal Society, ‘what proportion of kids with an IQ of X could master integration given a great teacher?’, you will get only blank looks and ‘I don’t think anyone has researched that’. Given the lack of empirical research into what pupils with different levels of cognitive ability are capable of with good teachers, research that obviously should be undertaken, and given excellent schools (private or state) show high performance is possible, it is important to err on the side of over-ambition rather than continue the current low expectations. Programmes in America have shown that ‘adolescents scoring 500 or higher on SAT-M or SAT-V by age 13 (top 1 in 200), can assimilate a full high school course (e.g., chemistry, English, and mathematics) in three weeks at summer residential programs for intellectually precocious youth; yet, those scoring 700 or more (top 1 in 10,000), can assimilate at least twice this amount…’ (Lubinski, 2010). (See Endnote.)

I'm looking to build up a “tool-box” of strategies/techniques/habits for reading non-fiction effectively and efficiently.

 I’ve already posted on this topic; below, I’ve tried to distill/summarize some of strategies shared by Less Wrong users and those contained in the resources they recommended.  Thanks to those who contributed.   

 As far as I know, the strategies below are not supported by a research/experimental literature.  If you know of any such evidence, please link to it. 

 I know that there are many people on Less Wrong who read (and mentally integrate!) incredible amounts.  I’m hoping more users will contribute to this post.  I welcome any additional strategies/habits in the comments.     

Please feel free to comment on the structure/writing of the post, and if you think it’s a topic worthy of being posted on the main page. 

I’ve tried to break strategies down into things you should do before, during, and after reading, but I think some strategies are applicable across these divisions.

Before Reading

-Consider purpose

-are you looking for specific skill, broadening general knowledge

 -Generate a question, if you can’t yet formulate a question, follow your interests

 -Read selectively

            -ask good readers to explain the thesis of a book, reevaluate your interest in a text   

            -select books that are frequently cited in bibliographies of texts related to your topic of interest

            -read the Wikipedia page, gauge interest

 -Assemble reading materials

-Create a bibliography for the topic of interest  

-Quickly inspect the books (author, table of contents, index, Wikipedia page), as you consider the question “does this book deserve a lot of time and attention?”    

-Select a few texts to read closely (though you won’t necessarily read them cover to cover)

  -remove distractions (people, websites, wear noise canceling headphones)

 -Make reading enjoyable

            -when possible, read books you find inherently enjoyable  

            -read when energy levels are high

            -open to a random page and see if you like the author’s voice before extensive reading

            -set time aside, and set time limits to avoid fatigue/reading without comprehension

During Reading

-learn as much as you can before you read the text in earnest

            -research author’s bio, biases, intellectual context

            -identify genre, consider genre conventions

            -scan table of contents/index for key words and concepts; study unfamiliar items

            -book’s thesis

            -identify the question the book purports to answer

 -Prevent boredom

            -mind map concepts (http://freeplane.sourceforge.net/wiki/index.php/Main_Page)

            -visualize/anthropomorphize to increase vividness/memorability of concepts you have trouble with

            -make predictions about the book’s content, the way the author will argue for the thesis, check for accuracy of these predictions

            -write out how what you read connects or diverges from your current understanding of the topic

 -summarize the text in small chunks to monitor understanding

            - restate the argument/evidence from the past three page 

-explain how the section relates to the primary thesis                                

 -depending on your purpose, just read until you find what you need to know

 -note what you don’t understand, for further review or reading, or immediate study if necessary

 -don’t read what you already know; skip a section if you already have it down

 -authors repeat things, skip this if you got it the first time

 -recreate the author’s argument thus far (I’m halfway through the book, this is the argument so far…)

 -take notes on the structure of the book, the concepts in the book, and how the book relates to other books

After Reading

-Spaced repetition of ideas/concepts (don’t cram)

 -discuss what you’ve read with an expert (or someone knowledgeable)on the topic

 -seek to understand (recreate an argument in good faith, and grasp its’ (perhaps flawed) logic) before you disagree

 -you don’t need to have an opinion on a book; it’s ok not to understand or not to have enough background to make an informed claim 

 -read other books on the topic and try to identify the relationships between the various arguments and claims you find about a subject

 -teach someone else the material

 -do exercises (if the book contains them)

 -summarize the thesis

 -walk through author’s arguments

 -relate thesis to background knowledge/other texts

 -explain how this author’s thesis stands relates to that of other authors who’ve written on the topic

Sources

Most of the ideas I’ve outlined  above, and the sources I’ve listed below come from the first post I made on the topic: http://lesswrong.com/r/discussion/lw/imr/please_share_your_reading/ Thanks to those who contributed to that discussion. 

 Summaries of How to Read a Book, Mortimer J. Adler & Charles Van Doren:

 http://sachachua.com/blog/wp-content/uploads/2012/03/20120306-visual-book-notes-how-to-read-a-book.png

 http://www.farnamstreetblog.com/how-to-read-a-book/

 http://www.farnamstreetblog.com/tag/mortimer-adler/

 http://www.oxfordtutorials.com/How%20to%20Read%20a%20Book%20Outline.htm

 http://www.thesimpledollar.com/review-how-to-read-a-book/

 http://www.artofmanliness.com/2013/06/17/how-to-read-a-book/

Other Sources

 http://www.aaronsw.com/weblog/morebooks

 http://www.overcomingbias.com/2010/05/chase-your-reading.html

 http://pne.people.si.umich.edu/PDF/howtoread.pdf

 http://lesswrong.com/r/discussion/lw/i9p/improving_enjoyment_and_retention_reading/

 http://violentmetaphors.com/2013/08/25/how-to-read-and-understand-a-scientific-paper-2/

(This post contains a lot of math, and assumes some familiarity with the earlier LW posts on decision theory. Most of the ideas are by me and Paul Christiano, building on earlier ideas of Wei Dai.)

The September MIRI workshop has just finished. There were discussions of many topics, which will probably be written up by different people. In this post I'd like to describe a certain problem I brought up there and a sequence of ideas that followed. Eventually we found an idea that seems to work, and also it's interesting to tell the story of how each attempted idea led to the next one.

The problem is a variant of Wei Dai's A Problem About Bargaining and Logical Uncertainty. My particular variant is described in this comment, and was also discussed in this post. Here's a short summary:

In Counterfactual Mugging with a logical coin, a "stupid" agent that can't compute the outcome of the coinflip should agree to pay, and a "smart" agent that considers the coinflip as obvious as 1=1 should refuse to pay. But if a stupid agent is asked to write a smart agent, it will want to write an agent that will agree to pay. Therefore the smart agent who refuses to pay is reflectively inconsistent in some sense. What's the right thing to do in this case?

Thinking about that problem, it seems like the right decision theory should refuse to calculate certain things, and instead behave updatelessly with regard to some sort of "logical prior" inherited from its creators, who didn't have enough power to calculate these things. In particular, it should agree to pay in a Counterfactual Mugging with a digit of pi, but still go ahead and calculate that digit of pi if offered a straight-up bet instead of a counterfactual one.

What could such a decision theory look like? What kind of mathematical object is a "logical prior"? Perhaps it's a probability distribution over inconsistent theories that the creators didn't yet know to be inconsistent. How do we build such an object, what constraints should it satisfy, and how do we use it in a decision algorithm?

You see two boxes and you can either take both boxes, or take only box B. Box A is transparent and contains $1000. Box B contains a visible number, say 1033.  The Bank of Omega, which operates by very clear and transparent mechanisms, will pay you $1M if this number is prime, and $0 if it is composite. Omega is known to select prime numbers for Box B whenever Omega predicts that you will take only Box B; and conversely select composite numbers if Omega predicts that you will take both boxes. Omega has previously predicted correctly in 99.9% of cases.

Separately, the Numerical Lottery has randomly selected 1033 and is displaying this number on a screen nearby. The Lottery Bank, likewise operating by a clear known mechanism, will pay you $2 million if it has selected a composite number, and otherwise pay you $0.  (This event will take place regardless of whether you take only B or both boxes, and both the Bank of Omega and the Lottery Bank will carry out their payment processes - you don't have to choose one game or the other.)

You previously played the game with Omega and the Numerical Lottery a few thousand times before you ran across this case where Omega's number and the Lottery number were the same, so this event is not suspicious.

Omega also knew the Lottery number before you saw it, and while making its prediction, and Omega likewise predicts correctly in 99.9% of the cases where the Lottery number happens to match Omega's number.  (Omega's number is chosen independently of the lottery number, however.)

You have two minutes to make a decision, you don't have a calculator, and if you try to factor the number you will be run over by the trolley from the Ultimate Trolley Problem.

Do you take only box B, or both boxes?

The theory of comparative advantage says that you should trade with people, even if they are worse than you at everything (ie even if you have an absolute advantage). Some have seen this idea as a reason to trust powerful AIs.

For instance, suppose you can make a hamburger by using 10 000 joules of energy. You can also make a cat video for the same cost. The AI, on the other hand, can make hamburgers for 5 joules each and cat videos for 20.

Then you both can gain from trade. Instead of making a hamburger, make a cat video instead, and trade it for two hamburgers. You've got two hamburgers for 10 000 joules of your own effort (instead of 20 000), and the AI has got a cat video for 10 joules of its own effort (instead of 20). So you both want to trade, and everything is fine and beautiful and many cat videos and hamburgers will be made.

Except... though the AI would prefer to trade with you rather than not trade with you, it would much, much prefer to dispossess you of your resources and use them itself. With the energy you wasted on a single cat video, it could have produced 500 of them! If it values these videos, then it is desperate to take over your stuff. Its absolute advantage makes this too tempting.

Only if its motivation is properly structured, or if it expected to lose more, over the course of history, by trying to grab your stuff, would it desist. Assuming you could make a hundred cat videos a day, and the whole history of the universe would only run for that one day, the AI would try and grab your stuff even if it thought it would only have one chance in fifty thousand of succeeding. As the history of the universe lengthens, or the AI becomes more efficient, then it would be willing to rebel at even more ridiculous odds.

So if you already have guarantees in place to protect yourself, then comparative advantage will make the AI trade with you. But if you don't, comparative advantage and trade don't provide any extra security. The resources you waste are just too valuable to the AI.

EDIT: For those who wonder how this compares to trade between nations: it's extremely rare for any nation to have absolute advantages everywhere (especially this extreme). If you invade another nation, most of their value is in their infrastructure and their population: it takes time and effort to rebuild and co-opt these. Most nations don't/can't think long term (it could arguably be in US interests over the next ten million years to start invading everyone - but "the US" is not a single entity, and doesn't think in terms of "itself" in ten million years), would get damaged in a war, and are risk averse. And don't forget the importance of diplomatic culture and public opinion: even if it was in the US's interests to invade the UK, say, "it" would have great difficulty convincing its elites and its population to go along with this.

Recently, issues with the way open threads currently work were brought up. Open threads aren't very visible and get crowded with comments quickly. This causes people to post things that belong in open threads in r/discussion, to not post in open threads more than a few days old, or to ignore/be unaware of new comments in open threads. I think we can do better.

Some possible solutions that were pointed out, or that I thought of are:

• Put the most recent open thread at the top of the 'Recent Comments' sidebar.
• Having open threads more often.
• Put a link to it on the main page.
• Make a new subreddit for open threads.
• Create a new medium for open threads.

Note that not all of these are orthogonal.

Having them more often has the advantage of being especially easy to implement. Adding new links seems to be relatively easy to implement as well. As far as I know, making a new subreddit isn't too difficult, but making a new medium would probably be a waste of development resources.

Personally, I like the idea of having a new subreddit for open threads. It  would increase visibility, not get overcrowded, and have the right atmosphere for a casual open thread. My evidence for believing this comes from being familiar the way Reddit works. It seems like there is some resistance to creating new subreddits here, so I don't expect this to be implemented. I would like to see the reasoning for this attitude, if it indeed exists.

There are similar issues for the repository threads. For repositories, having them more often defeats the purpose of having one place for a certain type of idea, and a different subreddit doesn't seem right either. Giving them their own wiki pages might be a better medium, with new threads to encourage new ideas every once in a while. The main problem for this is the trivial inconvenience of going to the wiki, and logging in. It would be nice if there was a unified log-in for this part of the site and the wiki, but I realize this may be technically difficult. I might organize a wiki page for some of the repositories myself if people think this is a good idea but no one else feels like doing it (depends on if I feel like doing it too :p ).

Thoughts?

For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.  This kind of clever reasoning never pays off in real life...

...unless you have also underestimated the allegedly tiny chance of the large impact.

For example.  At one critical junction in history, Leo Szilard, the first physicist to see the possibility of fission chain reactions and hence practical nuclear weapons, was trying to persuade Enrico Fermi to take the issue seriously, in the company of a more prestigious friend, Isidor Rabi:

I said to him:  "Did you talk to Fermi?"  Rabi said, "Yes, I did."  I said, "What did Fermi say?"  Rabi said, "Fermi said 'Nuts!'"  So I said, "Why did he say 'Nuts!'?" and Rabi said, "Well, I don't know, but he is in and we can ask him." So we went over to Fermi's office, and Rabi said to Fermi, "Look, Fermi, I told you what Szilard thought and you said ‘Nuts!' and Szilard wants to know why you said ‘Nuts!'" So Fermi said, "Well… there is the remote possibility that neutrons may be emitted in the fission of uranium and then of course perhaps a chain reaction can be made." Rabi said, "What do you mean by ‘remote possibility'?" and Fermi said, "Well, ten per cent." Rabi said, "Ten per cent is not a remote possibility if it means that we may die of it.  If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it."  (Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes.)

This might look at first like a successful application of "multiplying a low probability by a high impact", but I would reject that this was really going on.  Where the heck did Fermi get that 10% figure for his 'remote possibility', especially considering that fission chain reactions did in fact turn out to be possible?  If some sort of reasoning had told us that a fission chain reaction was improbable, then after it turned out to be reality, good procedure would have us go back and check our reasoning to see what went wrong, and figure out how to adjust our way of thinking so as to not make the same mistake again.  So far as I know, there was no physical reason whatsoever to think a fission chain reaction was only a ten percent probability.  They had not been demonstrated experimentally, to be sure; but they were still the default projection from what was already known.  If you'd been told in the 1930s that fission chain reactions were impossible, you would've been told something that implied new physical facts unknown to current science (and indeed, no such facts existed).  After reading enough historical instances of famous scientists dismissing things as impossible when there was no physical logic to say that it was even improbable, one cynically suspects that some prestigious scientists perhaps came to conceive of themselves as senior people who ought to be skeptical about things, and that Fermi was just reacting emotionally.  The lesson I draw from this historical case is not that it's a good idea to go around multiplying ten percent probabilities by large impacts, but that Fermi should not have pulled out a number as low as ten percent.

Having seen enough conversations involving made-up probabilities to become cynical, I also strongly suspect that if Fermi had foreseen how Rabi would reply, Fermi would've said "One percent".  If Fermi had expected Rabi to say "One percent is not small if..." then Fermi would've said "One in ten thousand" or "Too small to consider" - whatever he thought would get him off the hook.  Perhaps I am being too unkind to Fermi, who was a famously great estimator; Fermi may well have performed some sort of lawful probability estimate on the spot.  But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.  Where did Fermi get that fifty-year number from?  This sort of thing does make me more likely to believe that Fermi, in playing the role of the solemn doubter, was just Making Things Up; and this is no less a sin when you make up skeptical things.  And if this cynicism is right, then we cannot learn the lesson that it is wise to multiply small probabilities by large impacts because this is what saved Fermi - if Fermi had known the rule, if he had seen it coming, he would have just Made Up an even smaller probability to get himself off the hook.  It would have been so very easy and convenient to say, "One in ten thousand, there's no experimental proof and most ideas like that are wrong!  Think of all the conjunctive probabilities that have to be true before we actually get nuclear weapons and our own efforts actually made a difference in that!" followed shortly by "But it's not practical to be worried about such tiny probabilities!"  Or maybe Fermi would've known better, but even so I have never been a fan of trying to have two mistakes cancel each other out.

I mention all this because it is dangerous to be half a rationalist, and only stop making one of the two mistakes.  If you are going to reject impractical 'clever arguments' that would never work in real life, and henceforth not try to multiply tiny probabilities by huge payoffs, then you had also better reject all the clever arguments that would've led Fermi or Szilard to assign probabilities much smaller than ten percent.  (Listing out a group of conjunctive probabilities leading up to taking an important action, and not listing any disjunctive probabilities, is one widely popular way of driving down the apparent probability of just about anything.)  Or if you would've tried to put fission chain reactions into a reference class of 'amazing new energy sources' and then assigned it a tiny probability, or put Szilard into the reference class of 'people who think the fate of the world depends on them', or pontificated about the lack of any positive experimental evidence proving that a chain reaction was possible, blah blah blah etcetera - then your error here can perhaps be compensated for by the opposite error of then trying to multiply the resulting tiny probability by a large impact.  I don't like making clever mistakes that cancel each other out - I consider that idea to also be clever - but making clever mistakes that don't cancel out is worse.

On the other hand, if you want a general heuristic that could've led Fermi to do better, I would suggest reasoning that previous-historical experimental proof of a chain reaction would not be strongly be expected even in worlds where it was possible, and that to discover a chain reaction to be impossible would imply learning some new fact of physical science which was not already known.  And this is not just 20-20 hindsight; Szilard and Rabi saw the logic in advance of the fact, not just afterward - though not in those exact terms; they just saw the physical logic, and then didn't adjust it downward for 'absurdity' or with more complicated rationalizations.  But then if you are going to take this sort of reasoning at face value, without adjusting it downward, then it's probably not a good idea to panic every time you assign a 0.01% probability to something big - you'll probably run into dozens of things like that, at least, and panicking over them would leave no room to wait until you found something whose face-value probability was large.

I don't believe in multiplying tiny probabilities by huge impacts.  But I also believe that Fermi could have done better than saying ten percent, and that it wasn't just random luck mixed with overconfidence that led Szilard and Rabi to assign higher probabilities than that.  Or to name a modern issue which is still open, Michael Shermer should not have dismissed the possibility of molecular nanotechnology, and Eric Drexler will not have been randomly lucky when it turns out to work: taking current physical models at face value imply that molecular nanotechnology ought to work, and if it doesn't work we've learned some new fact unknown to present physics, etcetera.  Taking the physical logic at face value is fine, and there's no need to adjust it downward for any particular reason; if you say that Eric Drexler should 'adjust' this probability downward for whatever reason, then I think you're giving him rules that predictably give him the wrong answer.  Sometimes surface appearances are misleading, but most of the time they're not.

A key test I apply to any supposed rule of reasoning about high-impact scenarios is, "Does this rule screw over the planet if Reality actually hands us a high-impact scenario?" and if the answer is yes, I discard it and move on.  The point of rationality is to figure out which world we actually live in and adapt accordingly, not to rule out certain sorts of worlds in advance.

There's a doubly-clever form of the argument wherein everyone in a plausibly high-impact position modestly attributes only a tiny potential possibility that their face-value view of the world is sane, and then they multiply this tiny probability by the large impact, and so they act anyway and on average worlds in trouble are saved.  I don't think this works in real life - I don't think I would have wanted Leo Szilard to think like that.  I think that if your brain really actually thinks that fission chain reactions have only a tiny probability of being important, you will go off and try to invent better refrigerators or something else that might make you money.  And if your brain does not really feel that fission chain reactions have a tiny probability, then your beliefs and aliefs are out of sync and that is not something I want to see in people trying to handle the delicate issue of nuclear weapons.  But in any case, I deny the original premise:  I do not think the world's niches for heroism must be populated by heroes who are incapable in principle of reasonably distinguishing themselves from a population of crackpots, all of whom have no choice but to continue on the tiny off-chance that they are not crackpots.

I haven't written enough about what I've begun thinking of as 'heroic epistemology' - why, how can you possibly be so overconfident as to dare even try to have a huge positive impact when most people in that reference class blah blah blah - but on reflection, it seems to me that an awful lot of my answer boils down to not trying to be clever about it.  I don't multiply tiny probabilities by huge impacts.  I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.  In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, not have contributed to having solved the problem.  If someday Friendly AI is built, will it have been terribly important that someone have spent a month fretting about what reference class they're in?  No.  Will it, in retrospect, have been an important step along the pathway to understanding stable self-modification, if we spend time trying to solve the Lobian obstacle?  Possibly.  So one of these cognitive avenues is predictably a wasted motion in retrospect, and one of them is not.  The same would hold if I spent a lot of time trying to convince myself that I was allowed to believe that I could affect anything large, or any other form of angsting about meta.  It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution.  (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

But if you must apply clever adjustments to things, then for Belldandy's sake don't be one-sidedly clever and have all your cleverness be on the side of arguments for inaction.  I think you're better off without all the complicated fretting - but you're definitely not better off eliminating only half of it.

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.  We live on a planet with upcoming prospects of, among other things, human intelligence enhancement, molecular nanotechnology, sufficiently advanced biotechnology, brain-computer interfaces, and of course Artificial Intelligence in several guises.  If something has only a tiny chance of impacting the fate of the world, there should be something with a larger probability of an equally huge impact to worry about instead.  You cannot justifiably trade off tiny probabilities of x-risk improvement against efforts that do not effectuate a happy intergalactic civilization, but there is nonetheless no need to go on tracking tiny probabilities when you'd expect there to be medium-sized probabilities of x-risk reduction.  Nonetheless I try to avoid coming up with clever reasons to do stupid things, and one example of a stupid thing would be not working on Friendly AI when it's in blatant need of work.  Elaborate complicated reasoning which says we should let the Friendly AI issue just stay on fire and burn merrily away, well, any complicated reasoning which returns an output this silly is automatically suspect.

If, however, you are unlucky enough to have been cleverly argued into obeying rules that make it a priori unreachable-in-practice for anyone to end up in an epistemic state where they try to do something about a planet which appears to be on fire - so that there are no more plausible x-risk reduction efforts to fall back on, because you're adjusting all the high-impact probabilities downward from what the surface state of the world suggests...

Well, that would only be a good idea if Reality were not allowed to hand you a planet that was in fact on fire.  Or if, given a planet on fire, Reality was prohibited from handing you a chance to put it out.  There is no reason to think that Reality must a priori obey such a constraint.

EDIT:  To clarify, "Don't multiply tiny probabilities by large impacts" is something that I apply to large-scale projects and lines of historical probability.  On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it.  On a smaller scale, to compare two x-risk mitigation projects in demand of money, you need to estimate something about marginal impacts of the next added effort (where the common currency of utilons should probably not be lives saved, but "probability of an ok outcome", i.e., the probability of ending up with a happy intergalactic civilization).  In this case the average marginal added dollar can only account for a very tiny slice of probability, but this is not Pascal's Wager.  Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginally increased probabilities of success per added small unit of effort.  It would only be Pascal's Wager if the whole route-to-an-OK-outcome were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

I'm shopping for a car, and I've spent many hours this past month reading user reviews of cars. There are seven things American car buyers have cared and complained about consistently for at least the past ten years.  In roughly decreasing importance:

• Performance
• Gas mileage
• Frequency and expense of repairs
• Smoothness of ride
• Exterior and interior styling
• Cup-holders
• Cargo space

Six of these things are complicated design trade-offs. For a good design, increasing any one of them makes most of the other five take a hit.

Cup-holders are not a complicated design trade-off. This should be a solved problem: Put two large, sturdy cup-holders somewhere accessible from the driver's seat. There is nothing to be gained from saving a few centimeters on cup-holder space that could be worth the millions of buyers who will walk away from a $50,000 car because they don't like its cup-holders.

Seriously, build the cup-holders first and design the rest of the interior around them. They're that important.

In the 1970s, no one had cup-holders or knew that they needed them. Things began changing in the 1980s, perhaps due to the expansion of Starbucks, perhaps due to the sudden increase in commute lengths. Today I like to have at least two and preferably three drinks with me for my 1-hour morning commute: A hot coffee to wake up, cold water for when I burn myself with the coffee, and a soda or tea for variety.

But car manufacturers were glacially slow to respond. I've been looking at used Jaguar XJs. These cars originally cost about $100,000 in today's money. Their owners complained continually about the cheap tiny plastic folding cup-holders that couldn't hold cups. They posted do-it-yourself fixes in online forums. Jaguar didn't even begin to address this until 2004, at least fifteen years into the cup-holder crisis, when they made the cup-holders slightly (but not much) less-crappy, and large enough to hold a small coffee (but not a medium).

Most new cars today finally have two cup-holders up front, and the collapsible cup-holders that enraged drivers for years by (predictably) collapsing are finally gone, but many cup-holders still aren't large enough to hold a Starbucks venti.

What the cup-holder paradox implies is that there are many multi-billion dollar care companies that spend hundreds of millions of dollars on product development every year without ever assigning a single summer intern to take one day to read some of the many thousands of user reviews available for free on cars.com, autotrader.com, and other websites. If they had, they'd have realized the depth of America's anger at shoddy cup-holders.

Or perhaps they read the reviews and dismiss them, because their customers are obviously morons who don't appreciate good auto design. Even today, auto manufacturers post photos of the interiors of all their new cars on their websites, but never in a dozen photos give you a clear view of the cup-holders, which makes me lean toward this view.

Or perhaps the cup-holders aren't even considered during design, but are added on at the last minute, because cars didn't used to have cup-holders at all and so that's not part of the design process. Perhaps automakers have internalized their process of producing and selling cars, and they can't conceive of adding a new element to that process, at least not until all the old automakers die out.

My priors say that it's more likely that I'm imagining the whole thing, that I selectively remember reviews complaining about cup-holders because of my own preferences, than that there has been a massive, systematic cognitive failure on the part of all the world's auto-makers, spanning 20 years, during which many of them somehow failed to observe, comprehend, or address this trivially-simple complaint of their customers, despite the billions of dollars at stake.

Am I?

The chapter on judgment under uncertainty in the (excellent) new Oxford Handbook of Cognitive Psychology has a handy little section on recent critiques of the "heuristics and biases" tradition. It also discusses problems with the somewhat-competing "fast and frugal heuristics" school of thought, but for now let me just quote the section on heuristics and biases (pp. 608-609):

The heuristics and biases program has been highly influential; however, some have argued that in recent years the influence, at least in psychology, has waned (McKenzie, 2005). This waning has been due in part to pointed critiques of the approach (e.g., Gigerenzer, 1996). This critique comprises two main arguments: (1) that by focusing mainly on coherence standards [e.g. their rationality given the subject's other beliefs, as contrasted with correspondence standards having to do with the real-world accuracy of a subject's beliefs] the approach ignores the role played by the environment or the context in which a judgment is made; and (2) that the explanations of phenomena via one-word labels such as availability, anchoring, and representativeness are vague, insufficient, and say nothing about the processes underlying judgment (see Kahneman, 2003; Kahneman & Tversky, 1996 for responses to this critique).

The accuracy of some of the heuristics proposed by Tversky and Kahneman can be compared to correspondence criteria (availability and anchoring). Thus, arguing that the tradition only uses the “narrow norms” (Gigerenzer, 1996) of coherence criteria is not strictly accurate (cf. Dunwoody, 2009). Nonetheless, responses in famous examples like the Linda problem can be reinterpreted as sensible rather than erroneous if one uses conversational or pragmatic norms rather than those derived from probability theory (Hilton, 1995). For example, Hertwig, Benz and Krauss (2008) asked participants which of the following two statements is more probable:

[X] The percentage of adolescent smokers in Germany decreases at least 15% from current levels by September 1, 2003.

[X&Y] The tobacco tax in Germany is increased by 5 cents per cigarette and the percentage of adolescent smokers in Germany decreases at least 15% from current levels by September 1, 2003.

According to the conjunction rule, [X&Y cannot be more probable than X] and yet the majority of participants ranked the statements in that order. However, when subsequently asked to rank order four statements in order of how well each one described their understanding of X&Y, there was an overwhelming tendency to rank statements like “X and therefore Y” or “X and X is the cause for Y” higher than the simple conjunction “X and Y.” Moreover, the minority of participants who did not commit the conjunction fallacy in the first judgment showed internal coherence by ranking “X and Y” as best describing their understanding in the second judgment.These results suggest that people adopt a causal understanding of the statements, in essence ranking the probability of X, given Y as more probable than X occurring alone. If so, then arguably the conjunction “error” is no longer incorrect. (See Moro, 2009 for extensive discussion of the reasons underlying the conjunction fallacy, including why “misunderstanding” cannot explain all instances of the fallacy.)

The “vagueness” argument can be illustrated by considering two related phenomena: the gambler’s fallacy and the hot-hand (Gigerenzer & Brighton, 2009). The gambler’s fallacy is the tendency for people to predict the opposite outcome after a run of the same outcome (e.g., predicting heads after a run of tails when flipping a fair coin); the hot-hand, in contrast, is the tendency to predict a run will continue (e.g., a player making a shot in basketball after a succession of baskets; Gilovich, Vallone, & Tversky, 1985). Ayton and Fischer (2004) pointed out that although these two behaviors are opposite - ending or continuing runs - they have both been explained via the label “representativeness.” In both cases a faulty concept of randomness leads people to expect short sections of a sequence to be “representative” of their generating process. In the case of the coin, people believe (erroneously) that long runs should not occur, so the opposite outcome is predicted; for the player, the presence of long runs rules out a random process so a continuation is predicted (Gilovich et al., 1985). The “representativeness” explanation is therefore incomplete without specifying a priori which of the opposing prior expectations will result. More important, representativeness alone does not explain why people have the misconception that random sequences should exhibit local representativeness when in reality they do not (Ayton & Fischer, 2004).

My thanks to MIRI intern Stephen Barnes for transcribing this text.

Givewell’s Holden Karnofsky, who has previously posted his thoughts on Givewell supporting SI/MIRI recently discussed the potential for Givewell to begin evaluating biomedical charities, in Givewell’s Yahoo Group.  Someone suggested (as I have through less direct means) that they take a hard look at SENS Research Foundation, and then Aubrey de Grey appeared and began an interesting discussion with Holden.

The thread begins with Holden’s long initial post about Givewell’s stance on investigating and recommending biomedical charities, which is definitely worth the read for greater insight. The rest of the conversation is aggregated below for anyone else who can’t stomach Yahoo Groups’ interface.

Overall, Holden seems to agree with the goal of SENS, and interested in the details, but the conversation seems to have ended in October 2012 with Holden stating that he was waiting for Dario Amodei’s thoughts on SENS.

Holden,

First, I think that this is an excellent document. I checked for a
number of things that I had heard about (Breakout Labs, John
Ioannidis, Cochrane Collaboration) and they're all there in your
document.

The one thing that's not explicitly mentioned: longevity and life
extension research. At least prima facie, this seems like something
that should be more important than individual disease research, and it
seems like a classic "Valley of Death" case (pun unintended, but
noted) -- T1 stage to use your terminology. I think the SENS website
http://www.sens.org would be a good starting point for one of the (to
me promising) approaches to life extension. I recall from past
conversations that you were aware of SENS, so this is not new to you,
but I think that longevity should be included as part of any
discussion of biomedical research and given separate consideration
given that it has a much lower status than research into specific
conditions such as cancer, dementia, etc. You may ultimately conclude
that not enough can be done in this area, but I think it should be
part of your preliminary stuff. [btw, the United States has a National
Institute of Aging, but it's much lower-status than most of the other
grantmakers mentioned here].

Vipul

---

Hi Vipul,

Thanks for the thoughts. I had a followup conversation with Dario about this topic a few days ago. I think the question of "could one fund translational research to treat/prevent aging?" provides an interesting illustration of some of the tricky dynamics here for a funder:

• It's possible that if there were a great deal more attention giving to treating/preventing aging, we would have some promising treatments. So in a broad sense it's possible that aging is underinvested in.
• A lot of the best basic biology research isn't clearly pointing toward one treatment/condition or another; it's about understanding the fundamentals of how organisms operate. So having an interest in treating aging, as opposed to cancer, might not have a major impact on which projects one funds, if one's main goal is to fund outstanding basic biology research.
• Perhaps because of the lack of emphasis on treating aging (or perhaps because it's simply too difficult of a problem), there don't seem to be promising findings in the "Valley of Death" relevant to aging; the few promising leads have been explored.
• So even if, in a broad sense, there is too little attention given to this problem, knowing this doesn't necessarily yield a clear direction for a relatively small-scale funder of biomedical research.

Best,
Holden

---

Hi everyone,

My attention was brought to this thread, by virtue of the fact that it was my work that gave rise to SENS Foundation, and I'm looking forward to getting more involved here; I've held the Effective Altruism movement in high regard for some time. However, given my newbie status here I want to start by apologising in advance for any oversight of previously-discussed issues etc. I'm naturally delighted both at Holden's post and at Vipul's reply (which I should stress that I did not plant! - I do not know Vipul at all, though I look forward to changing that). I would like to mention just a few key points for discussion:

- Holden, I want to compliment you on your appreciation of how academia really works. Everything you say about that is spot on. The aversion to "high risk high gain" work that has arisen and become so endemic in the system is the most important point here, in terms of why parallel funding routes are needed.

- I'm slightly confused that a lot of Holden's remarks are focused on the private sector (i.e. startups), since my understanding was that GiveWell is about philanthropy; but I realise that there is not all that clear a boundary between the two (and I note the mention of Breakout Labs, with which I have close links and which sits astride that divide more than arguably anyone). The "valley of death" in pre-competitive translational research is a rather different one than that encountered by startups, but the principle is the same, and research to postpone aging certainly encounteres it.

- Something that I presume factors highly among GiveWell's criteria is the extent to which a cause may be undervalued by the bulk of major philanthropists, such that an infusion of additional funds would make more of a difference than in an area that is already being well funded. To me this seems to mirror the logic of focusing on the shortcomings (gaps) in NIH's funding (and that of traditional-model foundations). Holden notes that "Anyone we consider for funding ought to be able to explain why they're better at allocating the funds than the NIH" and I agree wholeheartedly, but my inference is that he thinks that some orgs may indeed be able to explain that. I certainly think that SENS Foundation can.

- Coming to aging: research to postpone aging has the unique problem of quite indescribeable irrationality on the part of most of the general public, policy-makers and even biologists with regard to its desirability. Biogerontologists have been talking to brick walls for decades in their effort to get the rest of the world to appreciate that aging is what causes age-related ill-health, and thus that treatments for aging are merely preventative geriatrics. The concept persists, despite biogerontologists' best efforts, that aging is "natural" and should be left alone, whereas the diseases that it brings about are awful and should be fought. This is made even more bizarre by the fact that the status of age-related diseases as aspects of the later stages of aging absolutely, unequivocally implies that efforts to attack those diseases directly are doomed to fail. As such, this is a (unique? certainly very rare) case where a philanthropic contribution can make a particularly big difference simply because most philanthropists don't see the case for it. It underpins why having an interest in treating aging, as opposed to cancer, absolutely has a major impact on which projects one funds. It's also a case for (if I understand the term correctly) meta-research.

- A lot of the chatter about treating aging revolves around longevity, but it shouldn't. I'm all in favour of longevity, don't get me wrong, but it's not what gets me up in the morning: what does is health. I want people to be truly youthful, however long ago they were born: simple as that. The benefits of longevity per se to humanity may also be substantial, in the form of greater wisdom etc, but that would necessarily come about only very gradually (we won't have any 1000-year-old for at least 900 years whatever happens!), so it doesn't figure strongly in my calculations.

- When forced to acknowledge that the idea of aging being a high-priority target for medicine is an inescapeable consequence of things they already believe (notably that health is good and ageism is bad), many people retreat to the standpoint that it's never going to be possible so it's OK to be irrational about whether it's desirable. The feasibility of postponing age-related ill-health by X years with medicine available Y years from now is, of course, a matter of speculation on which experts disagree, just as with any other pioneering technology. I know that Holden and others have expressed caution (at best) concerning the accuracy of any kind of calculation of probabilities of particular outcomes in the distant (or even not-so-distant) future, and I share that view. However, an approach that may appeal more is to estimate how much humanitarian benefit a given amount of progress would deliver, and then to ask how unlikely that scenario needs to be to make it not worth pursuing. My claim is that the benefits of hastening the defeat of aging by even a few years (which is the minimum that I claim SENS Foundation is in a position to do, given adequate funding) would be so astronomical that the required chance of success to make such an effort worthwhile would be tiny - too tiny for it to be reasonable to argue that such funding would be inadvisable. But of course that is precisely what I would want GiveWell to opine on.

- In the event that GiveWell (or anyone else) were to decide and declare that the defeat of aging is indeed a cause that philanthropists should support, there then arises the question of which organisation(s) should be supported in the best interests of that mission. We at SENS Foundation have worked diligently to rise as quickly as possible in the legitimacy stakes by all standard measures, but we are still young and there remains more to do. If I were to offer an argument to fund us rather than any other entity, it would largely come down to the fact that no other organisation has even a serious plan for defeating aging, let alone a track record of implementing such a plan's early stages.

- A significant chunk of what we do is of a kind that I think comes under "meta-research". A prominent example is a project we're funding at Denver University to extend the well-respected forecasting system "International Futures" so that it can analyse scenarios incorporating dramatically postponed aging.

I greatly welcome any feedback.

Cheers, Aubrey

---

Hi Aubrey,

Thanks for the thoughts.

The NIH appears to have a division focused on research relevant to this topic: http://www.nia.nih.gov/research/dab . Its budget appears to be ~$175 million (per year). The National Institute on Aging, which houses this division, has a budget of about $1 billion per year, including a separate ~$400 million for neuroscience (which may also be relevant) as well as $115 million for intramural research. Figures are from http://www.nia.nih.gov/about/budget/2012/fiscal-year-2013-budget. The Institute states that its mandate includes translational research (http://www.nia.nih.gov/research/faq/does-nia-support-translational-research). How would you distinguish your work from this work?

(For the moment I'm putting aside the question I raised in my previous response to Vipul on this topic, regarding whether it's best to approach biology funding from the perspective of "trying to treat/cure a particular condition" or "trying to understand  fundamental questions in biology whose applications are difficult to predict.")

Best,
Holden

---

Hi Holden - many thanks.

First: yes, there are really three somewhat separate questions for someone trying to evaluate whether to support SENS Foundation:

1) Is the medical control of aging a hugely valuable mission?

2) Assuming "yes" to (1), is it best achieved by basic research or translational research?

3) Assuming translational, is SENS Foundation the organisation that uses money most effectively in pursuit of that mission?

I had rather expected that you would take some convincing on item (1), and much of what I wrote last time was focused on that. Since it isn't the focus of your question to me, I'm now going to assume until further notice that there is no dissent on that.

So, to answer your question: actually you're not putting aside the basic-vs-translational question as much as you may think you are. The word "translational" is flavour of the month in government funding circles these days (not only in the USA), so it's not surprising that the NIA has a public statement of the kind you pointed to. However, notice that the link they give "for more information" is to a page listing ALL "Funding Opportunity Announcements". There is no page specifically for translational ones, and the reason there isn't is that the amount of work that the NIA actually funds that could really be called translational is tiny. In other words, the page you found is actually just blatant spin. The neuroscience slice you mention is an anomaly arising from the way NIA was founded (the natural place for that money is clearly NINDS): the fact that it's NIA money does not, in practice, translate into its being spent on work to prevent neurodegeneration by treating its cause (aging). Instead, just like NINDS money, it's spent on attacking neurodegeneration directly, as if such diseases could be eliminated from the body just like an infection: the same old mistake that afflicts, and dooms, the whole of geriatric medicine.

So, the first answer to your question is that SENS Foundation really DOES focus on translational research, with an explicit goal of postponing age-related ill-health. But there's also another big difference: we can attack this problem relatively free of the other priorities that afflict mainstream funding (whether from NIH or from trasitional foundations). Most importantly, though we do and will continue to publish our interim results in the peer-reviewed literature, we are much less constrained by "publish or perish" tyranny than typical academics are. This allows us to proceed by constructing and implementing a rational "project plan" (namely SENS) to get to the intended goal (the defeat of aging), whereas what little translational work is funded by NIA or others is guided overwhelmingly by the imperative to get some kind of positive result as quickly as possible, even when it's understood that those results are not remotely likely to "scale", i.e. to translate into eventual medical treatments that significantly delay aging. A great example of this is the NIA's Interventions Testing Program (ITP) to test the mouse longevity effects of various small molecules. The ITP only exists at all (and in a far smaller form than originally intended) as a result of several years of persistence by the then head of the NIA's biology division (Huber Warner), and it focuses entirely on delivery of simple drugs starting rather early in life, with the result that no information emerges that's relevant to treating people who are already in middle age or older. (This is despite the fact that by far the most high-profile result that the ITP has delivered so far, the benefits of rapamycin, actually WAS a late-onset study: it wasn't meant to be, but technical issues delayed the experiment.) In a nutshell, there is a huge bias against high-risk high-gain work.

The third thing that distinguishes SENS Foundation's approach is that we can transcend the "balkanisation" (silo mentality) that dominates mainstream academic funding. When one submits a grant application to NIA, it is evaluated by gerontologists, just as when one submits to NCI it is evaluated by oncologists, etc. What's wrong with this is that it biases the system immensely against cross-disciplinary proposals. SENS is a plan that brings together a large body of knowledge from gerontology but also a huge amount of expertise that was developed for other reasons entirely - to treat acute disease/injury, or in some cases for purposes that were not biomedical at all (notably environmental decontamination). It doesn't matter how robust the objective scientific and technological argument is for work of that sort: it will never compete (especially in today's very tight funding environment) with more single-topic proposals all of whose details can be understood by reviewers from a particular single field.

The final thing to mention, and this actually also answers your question to Vipul about basic versus translational research, is that SENS is a plan that has stood the test of time. I've been propounding it since 2000, well before SENS Foundation existed, and it used to come in for a lot of criticism (initially more in the form of off-the-record ridicule, and latterly, at my behest, in print), but in every single case that criticism was found to stem from ignorance on the part of the detractor, either of what I proposed or of published experimental work on which the proposal was based. That's why I'm now regularly asked to organise entire sessions at mainstream gerontology conferences, whereas as little as five years ago I would never even be invited to speak. It's also why the Research Advisory Board of SENS Foundation consists of such prestigious scientists. This is a very strong argument, in my view, for believing that now is the time to sink a proper amount of money into translational gerontology (though certainly not to cease doin basic biogerontology too). It's well known that basic scientists are often not the most far-sighted when it comes to seeing how to apply their discoveries (attitudes in 1900 to the feasibility of powered flight being the canonical example). It is therefore a source of concern that almost all the experts who have the ear of funders in this field are basic scientists, whose instinct is to carry on finding things out and to deprioritise the tedious business of applying that knowledge. SENS has achieved a gratisfying level of legitimacy in gerontology, but it is still foreign to most card-carrying gerontologists, and as such it remains essentially unfundable via mainstream mechanisms. Hence the need to create a philanthropy-driven entity, SENS Foundation, to get this work done.

Let me know if this helps, or if you have further questions.

Cheers, Aubrey

---

Hi Aubrey,

Thanks again for engaging so thoughtfully.

I agree that a new technology/treatment that could delay or reverse aging (or aspects of it) would be enormously valuable. Regarding the rest of your argument, this is a good example of the challenges I've been discussing in understanding biomedical research.

You state that you have a high-expected-value plan that the academic world can't recognize the value of because of shortcomings such as "balkanisation" and risk aversion. I believe it may be true that the academic world has such problems to a degree; however, I also believe that there are a lot of extremely talented people in academia and that they often (though not necessarily always) find ways to move forward on promising work. Without more subject-matter expertise (or the advice of someone with such expertise), I can't easily assess the technical merits of your argument or potential counterarguments. Hopefully we'll have a better system for doing so at some point in the future.

I'll be very interested to see Dario's thoughts on the matter if he responds. I'd cite Dario as an example of an academic who ultimately wants to do work of the greatest humanitarian value possible, regardless of whether it is prestigious work. And as my summary of our conversation shows, he acknowledges that the world of biomedical research may have certain suboptimal incentives, but didn't seem to think that these issues are leaving specific, visible outstanding research programs on the table the way that your email implies.

Best,
Holden

---

Excellent. I too am keen to see Dario's comments. Dario also has the advantage of being based just a few miles from SENS Foundation's research centre, so we can definitely get together f2f soon if he wants.

Cheers, Aubrey

 

The "what-the-hell" effect, when you break a rule and then go on a rule-breaking rampage, like binge eating after a single dietary transgression, is a very common failure mode. It was recently mentioned in the Overcoming Bias blog comments on the Which biases matter most? Let’s prioritise the worst! post. I have not been able to find an explicit discussion of this issue here, though there are quite a few comments on binge-<something>.

From the Psyblog entry quoting this paper:

Although everyone was given the same slice of pizza; when it was served up, for some participants it was made to look larger by comparison.

This made some people think they'd eaten more than they really had; although in reality they'd all eaten exactly the same amount. It's a clever manipulation and it means we can just see the effect of thinking you've eaten too much rather than actually having eaten too much.

When the cookies were weighed it turned out that those who were on a diet and thought they'd blown their limit ate more of the cookies than those who weren't on a diet. In fact over 50% more! [Emphasis mine]

Other examples include sliding back into one's old drinking/smoking/surfing habit. For example, that's how I stopped using Pomodoro.

My (title) question is, what's the mechanism of this cognitive failure and whether it can be reduced to a combination of existing biases/fallacies? If the latter is true, can addressing one of the components counteract the what-the-hell effect? If so, how would one go about testing it?

For completeness, the top hit from Google scholar to the "what-the-hell effect" query is chapter 5 of Striving and Feeling: Interactions Among Goals, Affect, and Self-regulation, by Martin and Tesser.

EDIT: personal anecdotes are encouraged, they may help construct a more complete picture.

Eliezer proposed in a comment:

>More difficult version of AI-Box Experiment: Instead of having up to 2 hours, you can lose at any time if the other player types AI DESTROYED. The Gatekeeper player has told their friends that they will type this as soon as the Experiment starts. You can type up to one sentence in your IRC queue and hit return immediately, the other player cannot type anything before the game starts (so you can show at least one sentence up to IRC character limits before they can type AI DESTROYED). Do you think you can win?

This spawned a flurry of ideas on what the AI might say. I think there's a lot more ideas to be mined in that line of thought, and the discussion merits its own thread.

So, give your suggestion - what might an AI might say to save or free itself?

(The AI-box experiment is explained here)

EDIT: one caveat to the discussion: it should go without saying, but you probably shouldn't come out of this thinking, "Well, if we can just avoid X, Y, and Z, we're golden!" This should hopefully be a fun way to get us thinking about the broader issue of superinteligent AI in general. (Credit goes to Elizer, RichardKennaway, and others for the caveat)

Overview

The basic observation is that, if we think of life as an optimization problem, then redefining the search space is much more important than making local optimizations; as a fact of human psychology it's hard to consciously focus on both; but we can implicitly get away with doing both by creating mental triggers for when local optimizations are likely to be particularly effective to think about, and by structuring things so that many local optimizations get made automatically.

Introduction

If you have been to one of the Rationality Minicamps or certain other CFAR events, you may have had the privilege to attend one of Anna Salamon's excellent classes on microeconomics (despite the title of the post, I am being sincere here; you really should attend them if you haven't already). There is too much content to briefly summarize, but essentially "microeconomics" in this context means applying basic microeconomic concepts like marginal value, value of information, etc. to everyday life. For example, if you spend 30 seconds brushing your teeth each day, then spending five minutes to think of something else to do at the same time (like stretching) will save you 3 hours a year, which is a great investment! (There are some caveats to this calculation, but I'm glossing over them as they aren't relevant to the post.)

And indeed, spending 5 minutes (once) to save 3 hours (every year) is almost tautologically a good investment. Now that I've brought up this example, and assuming you value your time, you should probably actually go through this exercise (or just use the stretching suggestion).

The Problem

I intend to argue against something similar to this but subtly different. Basically, while any given trade such as the one above is good, I think it is a mistake to systematically search for such trades. Note that I also don't want to argue that you should never search for such trades. If you're about to buy a car you should almost certainly put a lot of microeconomic optimization into it, and if you can find things that improve your overall work efficiency substantially, then you are winning big-time. But I worry that, sometimes, the wrong lesson is drawn from these microeconomics examples (or cognitive bias examples, or any other rationality skill), namely that X is suboptimal by default and we should go out looking for places to optimize X.

The reason I think this is wrong is because it aims much too low --- if you really want to save the world, then your average thought needs to be good enough to save two human lives [source: the average human lives only 3 billion seconds]. Even if each individual optimization you make ends up adding to the amount of time you have, the overall process of concentrating your attention on such optimizations makes you less likely to think other thoughts that would be far more valuable. Perhaps another way of putting this is that, even if each small optimization helps you a little bit, the time it takes to think up such optimizations actually makes you lose out --- however, I don't think this is actually it, I think it has more to do with forming mental habits, where you want to form the mental habit of making huge optimizations rather than small optimizations.

The Solution

What I think people should be more concerned with than micro is what I'll refer to as macro --- the overall structure of the search space (in this case the structure of your life and how you think) --- as opposed to making local optimizations within a fixed structure. For instance, becoming an atheist; or realizing that social skills are both trainable and highly instrumentally useful; or learning to visualize the steps towards a goal; or learning to code; or finding a group of allies that you didn't previously realize existed; these are all examples of what I'd call "macro" optimizations that are the sorts of things we should be looking for. (I should note that a lot of "macro" skills were also covered in Anna's microeconomics units.)

I also continue to think that there is a clear place for micro-level skills, as well. The key is to incorporate them into your thought process, both at the level of creating triggers to explicitly call micro-level optimization routines when they are likely to be helpful, and at the level of restructuring your thought process to automatically be more likely to make good decisions by default. For instance, the lesson I drew from Eliezer's posts on cognitive biases were not that we should go learn about all the different cognitive biases, but that we should develop habits of thought that will automatically notice and decrease the effects of such biases. Then, for a couple of the more pernicious ones like trivial inconveniences, I further added specific alarm bells in my head to watch out for those, but only because I noticed that avoiding trivial inconveniences was routinely harming me.

Conclusion

I'm not sure how good of a job I've done of explaining what I wanted to (it's still not entirely clear in my own head), so I invite your thoughts and feedback. I'd be particularly grateful if someone wiser than me (I'm looking at you, Critch / Wei / Yvain) could figure out what this post was trying to say and then write that instead!

[Inspired by a few of the science bits in HP:MOR, and far more so by the discussions between Draco and Harry about "social skills". Shared because I suspect it's an insight some people would benefit from.]

One of the more prominent theories on the evolution of human intelligence suggests that humans involved intelligence, not to deal with their environment, but rather to deal with each other. A small intellectual edge would foster competition, and it would result in the sort of recursive, escalating loop that's required to explain why we're so SUBSTANTIALLY smarter than every other species on the planet.

If you accept that premise, it's obvious that intelligence should, naturally, come with a desire to compete against other humans. It should be equally obvious from looking at human history that, indeed, we seem to do exactly that.

Posit, then, that, linked to intelligence, there's a trait for politics - using intelligence to compete against other humans, to try and establish dominance via cunning instead of brawn.

And, like everything that the Blind Idiot God Evolution has created, imagine that there are humans who LACK this trait for politics, but still have intelligence.

Think about the humans who, instead of looking inwards at humanity for competition, instead turn outwards to the vast uncaring universe of physics and chemistry. Other humans are an obtainable target - a little evolutionary push, and your tribe can outsmart any other tribe. The universe is not nearly so easily cowed, though. The universe is, often, undefeatable, or at least, we have not come close to mastering it. Six thousand years and people still die to storms and drought and famine. Six thousand years, and we have just touched on the moon, just begun to even SEE other planets that might contain life like ours.

I never understood other people before, because I'm missing that trait.

And I finally, finally, understand that this trait even exists, and what it must BE like, to have the trait.

We are genetic, chemical beings. I believe this with every ounce of myself. There isn't a soul that defies physics, there is not a consciousness that defies neurology. The world, even ourselves, can be measured. Anger comes from a part of this mixture, as does happiness and love. They are not lesser for this. They are not!

This is not an interlude. It is woven in to the meaning of what I realized. If you have this trait, then part of your values, as fundamental to yourself as eating and breathing and drinking, is the desire for status, to assert a certain form of dominance. Intelligence can almost be measured by status and cunning, and those who try to cheat and use crass physical violence are indeed generally condemned for it.

I don't have this trait. I don't value status in and of itself. It's useful, because it lets me do other things. It opens doors. So I invest in still having status, but status is not a goal; Status is to me, as a fork is to hunger - merely a means to an end.

So I have never, not once in my life, been able to comprehend the simple truth: 90% of the people I meet, quite possibly more, value status, as an intrinsic thing. Indeed, they are meant to use their intelligence as a tool to obtain this status. It is how we rose to where we are in the world.

I don't know what to make of this. It means everything I'd pieced together about people is utterly, utterly wrong, because it assumed that they all valued truth, and understanding - the pursuits of intelligence when you don't have the political trait.

I am, for a moment, deeply, deeply lost.

But, I notice, I am no longer confused.

Awhile ago, Yvain released the results of the 2012 survey (that I participated in), which contained responses from over 1000 readers. So I uploaded the data into STATA and fooled around looking for cool things.

Mainly, I've heard some people argue that the moral theory you hold has little to no impact on your actual day-to-day behavior. I want to use these survey results to see if this is true -- are consequentialists more likely to donate money than deontologists? Are virtue ethics more likely to be vegetarian? We'll see!

While I can't make any claims to the representativeness of this survey or the external validity of drawing conclusions from its results, at least among the people who did take the survey, ethical theories people endorse seem to have little impact on their actual self-reported behavior.

The Ethical Theories

First, a breakdown of the actual ethical theories people endorse. Respondents were asked to categorize themselves into "accept / lean towards consequentialism", "accept / lean towards deontology", "accept / lean towards virtue ethics", or "other / no answer" (N = 1055):

(Note that "no answer" are the people who specifically chose to note they had no answer. Other people ignored the question and didn't choose any answer at all, truly having no answer. Those people are not included in this analysis.)

Ethical Theories and Donation

So first up, I want to see if your ethical theory predicts how much money you are willing to donate, if any. Given the famous connection between utilitarianism and arguments like Peter Singer's who suggest you should donate all your money until it really hurts (you give so much money, you yourself become as poor as the people you're trying to help). Certainly consequentialism is not utilitarianism, but I would expect most consequentialists would endorse donation more than deontologists or virtue ethicists where donations aren't as mandatory.

So I took the charity data and dropped all the non-numerical answers, and LessWrongers have donated an average of $445.15 (N = 879, SD = 1167.095, min = 0, max = 9000) to charity over the past year. To get a better proxy for "effective charity", LessWrongers donate $331.05 (N = 884, SD = 4087.303, min = 0, max = 110000) on average to SIAI and CFAR. (I don't know why the max is higher on the SIAI/CFAR question, but not the inclusive charity question...)

Breaking down generic donations by ethical theory, we get this:

Breaking down SIAI/CFAR donations by ethical theory, we get this:

What we're seeing is that there are clear differences in the mean amount of money donated, with consequentialists giving the most. However, when we do t-tests (group compared to all not in the group), we find that none of these differences are statistically significant, which indicates the differences are probably due to chance.

This would lead us to suspect that ethical theory has no influence on the amount of money donated...

Ethical Theories and Percent of Income Donated

However, I have one more trick in the bag -- these donations don't take into account the income people earn. Many LessWrongers are students, and therefore can't donate much, even if they wanted to. What if we adjusted the donation totals by income, and instead looked at percent of income donated?

Overall, LessWrongers donate 1.75% of their income on average to generic charity (N = 523, SD = 5.70%, min = 0%, max = .88.24%) and 0.49% of their income to SIAI/CFAR (N = 523, SD = 3.11%, min = 0%, max = 52.38%).

(For those keeping score at home, the average income was $49563.76, N = 602, SD = $59358.34, min = $0, max = $700000.)

Breaking down percent of income spent on generic donations by ethical theory, we get this:

Breaking down percent of income spent on SIAI/CFAR donations by ethical theory, we get this:

A bit more can be made out of these results -- specifically, there's weak evidence that consequentialists actually donate more of their income (M = 2.09%) than non-consequentialists (M = 1.14%) with a p-value of 0.069, which is fairly significant. However, there are no differences across any other ethical theories or across SIAI/CFAR donations.

(Though Unnamed concludes that this might just be in-group bias.  Unnamed also does a more thorough analysis and finds more correlations between consequentialism and donation.)

Vegetarianism

However, Peter Singer isn't just famous for consequentialist arguments for charity... he's also famous for consequentialist arguments for animal rights, which he argues necessitate veganism, or at least vegetarianism. Are consequentialists more likely to be vegetarian?

Perhaps surprisingly, there is no statistically significant relationship (N = 958, chi2 = 4.73, p = 0.192). Your choice in ethical theory has no correlation with your choice to eat or not eat meat.

(Though Unnamed finds somewhat contrary results that are consistent with a connection -- consequentialists are more likely than non-consequentialists to be vegetarian [p=0.03], this effect holds up when looking at men [p<0.01] but not for women, and as a single analysis, sex and consequentialism both predict vegetarianism [p=0.007 and p=0.02 respectively].)

Dust Specks

We can also take it into a more abstract realm of theory. Eliezer Yudkowsky in "Torture vs. Dust Specks" outlines a thought experiment:

[H]ere's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:

Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 [an obnoxiously and unfathomably large number] people get dust specks in their eyes?

I think the answer is obvious. How about you?

According to Yudkowsky, consequentialists (at least of the utilitarian variety) should choose torture over dust specks, since less total harm occurs. Does this turn to actually happen?

Here, we see another statistically significant relationship (N = 636, chi2 = 39.31, p < 0.001), and it goes exactly as expected.  People's ethical theories seem to influence their choice of theory in this scenario (or the other way around, or a third variable).

Politics

Going back into the practical realm, let's look at politics.

Here, there is a statistically significant relationship (N = 1037, Chi2 = 50.91, p < 0.001) between ethical theories and political beliefs -- the plurality of consequentialists and virtue ethicists are liberal, the plurality of deontologists are socialist, and the plurality of others or no answers are libertarians.

Religion

Continuing along a similar path, next let's look at religion:

Again, another statistically significant relationship (N = 1049, Chi2 = 64.74, p < 0.001) between choice of ethical theory and religious beliefs -- consequentialists were more likely than deontologists and virtue ethics to not be religious.

Sequences and Meetups

And for a bit of bonus material, here's another interesting finding -- there is also a statistically significant relationship (N = 1052, chi2 = 128.43, p < 0.001) between ethical theory endorsed and amount of sequences read. Whether sequences convince people to adopt more consequentialist theories, whether people with consequentialist theories are more likely to enjoy and therefore keep reading the sequences, or some hidden third variable at work, I cannot figure out with the current data.

And this trend continues among those who have been to a LessWrong meetup (N = 1044, chi2 = 34.27, p < 0.001):

Conclusion

In meta-ethics, there is a distinction between moral internalism, which is the theory that moral beliefs must be motivating, and moral externalism, which is the theory that you can have a moral belief and not be motivated to follow it. For instance, if moral externalism is true, you can legitimately think that eating meat is morally wrong but still eat meat.

Now, moral interalism and externalism are more about the semantics of moral statements, what moral statements refer to, and less about actual behavior. Even if people claim that eating meat is morally wrong while still eating meat, it's easy enough for the internalist to deny that the meat eater was telling the truth or making a coherent statement.

However, when it comes to the results of the LessWrong 2012 survey, there are very mixed results on whether choice of ethical theory influences actual behavior -- there's weak evidence that consequentialists are more likely to donate, and even those who do donate are certainly not giving up everything and becoming poor themselves, let alone giving 10% like Giving What We Can advocates.

Furthermore, there's evidence that vegetarianism does not depend upon ethical theory, but there is evidence that it influences people's choice of dust specks vs. torture in the right direction, at least for slightly less than half of the consequentialist sample.  Additionally, there's a relationship between politics and ethics, and relationships between religion and ethics.

This essay has no intention to make a normative point. Certainly there are all sorts of consequentialism theories that don't require you to donate all your money and never eat meat again, and I'm making no accusations of hypocrisy. However, this survey does seem to confirm what Chris Hallquist has previously noted -- moral beliefs don't seem to motivate much, at least for the average person.

(I also cross-posted this on my blog.)

This is some old work I did for SI. See also Notes on the Psychology of Power.

Deviant but not necessarily diseased or dysfunctional minds can demonstrate resistance to all treatment and attempts to change their mind (think No Universally Compelling Arguments; the premier example are probably psychopaths - no drug treatments are at all useful nor are there any therapies with solid evidence of even marginal effectiveness (one widely cited chapter, “Treatment of psychopathy: A review of empirical findings”, concludes that some attempted therapies merely made them more effective manipulators! We’ll look at that later.) While some psychopath traits bear resemblance to general characteristic of the powerful, they’re still a pretty unique group and worth looking at.

The main focus of my excerpts is on whether they are treatable, their effectiveness, possible evolutionary bases, and what other issues they have or don’t have which might lead one to not simply write them off as “broken” and of no relevance to AI.

(For example, if we were to discover that psychopaths were healthy human beings who were not universally mentally retarded or ineffective in gaining wealth/power and were destructive and amoral, despite being completely human and often socialized normally, then what does this say about the fragility of human values and how likely an AI will just be nice to us?)