The argument that AIs provided with a reward channel will observe their controllers and learn to manipulate them is a valid one. Unfortunately, it's often framed in a way that feels counterintuitive or extreme, especially to AI designers. It typically starts with the standard reinforcement learning scenario, then posits that the AI becomes superintelligent and either manipulates the controller with super-social powers, or breaks out and gains control of its reward channel, killing or threatening its controllers.

And that is a fair argument. But conceptually, it leaps from a standard reinforcement learning scenario, to a science-fiction-sounding scenario. It might help to have intermediate scenarios: to show that even lower intelligence AIs might start exhibiting the same sort of behaviour, long before it gets to superintelligence.

So consider the following scenario. Some complex, trainable AI is tasked with writing automated news stories for a student newspaper. It trawls the web and composes its stories, then gets reward and feedback from the editors. Assume there are two editors for this newspaper, and they work on alternate days. The two editors have somewhat different ideas as to what constitutes a good story, so their feedbacks are different. After a while, the AI finds that it gets higher reward by using a certain style on Monday, Wednesday and Friday, and another style on Tuesday and Thursdays - this is a simple consequence of its reward mechanism.

But the rota isn't perfect. Sometime the Monday editor will edit a story so late on Monday that it's Tuesday, and sometimes the Tuesday editor will be up early and edit a story at the same time. This pattern is more common during the holidays, when the editors go to different time zones, often travel, and become more irregular in their editing.

So if the AI complex and skilled enough, then, simply through simple feedback, it will start building up a picture of its editors. It will figure out when they are likely to stick to a schedule, and when they will be more irregular. It will figure out the difference between holidays and non-holidays. Given time, it may be able to track the editors moods and it will certainly pick up on any major change in their lives - such as romantic relationships and breakups, which will radically change whether and how it should present stories with a romantic focus.

It will also likely learn the correlation between stories and feedbacks - maybe presenting a story define roughly as "positive" will increase subsequent reward for the rest of the day, on all stories. Or maybe this will only work on a certain editor, or only early in the term. Or only before lunch.

Thus the simple trainable AI with a particular focus - write automated news stories - will be trained, through feedback, to learn about its editors/controllers, to distinguish them, to get to know them, and, in effect, to manipulate them.

This may be a useful "bridging example" between standard RL agents and the superintelligent machines.

GiveWell have just announced their latest charity recommendations! What are everyone’s thoughts on them?

A summary: all of the old charities (GiveDirectly, SCI and Deworm the World) remain on the list. They're rejoined by AMF, as the room for more funding issues that led to it being delisted have been resolved to GiveWell's satisfaction. Together these organisations form GiveWell's list of 'top charities', which is now joined by a list of other charities which they see as excellent but not quite in the top tier. The charities on this list are Development Media International, Living Goods, and two salt fortification programs (run by GAIN and ICCIDD).

As normal, GiveWell's site contains extremely detailed writeups on these organisations. Here are some shorter descriptions which I wrote for Charity Science's donations page and my tool for donating tax-efficiently, starting with the new entries:

GiveWell's newly-added charities

Boost health and cognitive development with salt fortification

The charities GAIN and ICCIDD run programs that fortify the salt that millions of poor people eat with iodine. There is strong evidence that this boosts their health and cognitive development; iodine deficiency causes pervasive mental impairment, as well as stillbirth and congenital abnormalities such as severe retardation. It can be done very cheaply on a mass scale, so is highly cost-effective. GAIN is registered in the US and ICCIDD in Canada (although Canadians can give to either via Charity Science, which for complex reasons helps others who donate tax-deductibly to other charities), allowing for especially efficient donations from these countries, and taxpayers from other countries can also often give to them tax-deductibly. For more information, read GiveWell's detailed reviews of GAIN and ICCIDD.

Educate millions in life-saving practices with Development Media International

Development Media International (DMI) produces radio and television broadcasts in developing countries that tell people about improved health practices that can save lives, especially those of young children. Examples of such practices include exclusive breastfeeding. DMI are conducting a randomized controlled trial of their program which has found promising indications of a large decrease in children's deaths. With more funds they would be able to reach millions of people, due to the unparalleled reach of broadcasting. For more information, read GiveWell's detailed review.

Bring badly-needed goods and health services to the poor with Living Goods

Living Goods is a non-profit which runs a network of people selling badly-needed health and household goods door-to-door in their communities in Uganda and Kenya and provide free health advice. A randomized controlled trial suggested that this caused a 25% reduction in under-5 mortality among other benefits. Products sold range from fortified foods and mosquito nets to cookstoves and contraceptives. Giving to Living Goods is an exciting opportunity to bring these badly needed goods and services to some of the poorest families in the world. For more information, read GiveWell's detailed review.

GiveWell's old and returning charities

Treat hundreds of people for parasitic worms

Deworm the World and the Schistosomiasis Control Initiative (SCI) treat parasitic worm infections such as schistosomiasis, which can cause urinary infections, anemia, and other nutritional problems. For more information, read GiveWell's detailed review, or the more accessible Charity Science summary. Deworm the World is registered in the USA and SCI in the UK, allowing for tax-efficient direct donations in those countries, and taxpayers from other countries can also often give to them efficiently.

Make unconditional cash transfers with GiveDirectly

GiveDirectly lets you empower people to purchase whatever they believe will help them most. Eleven randomized controlled trials have supported cash transfers’ impact, and there is strong evidence that recipients know their own situation best and generally invest in things which make them happier in the long term. For more information, read GiveWell's detailed review, or the more accessible Charity Science summary.

Save lives and prevent infections with the Against Malaria Foundation

Malaria causes about a million deaths and two hundred million infections a year. Thankfully a $6 bednet can stop mosquitos from infecting children while they sleep, preventing this deadly disease. This intervention has exceptionally robust evidence behind it, with many randomized controlled trials suggesting that it is one of the most cost-effective ways to save lives. The Against Malaria Foundation (AMF) is an exceptional charity in every respect, and was GiveWell's top recommendation in 2012 and 2013. Not all bednet charities are created equal, and AMF outperforms the rest on every count. They can distribute nets cheaper than most others, for just $6.13 US. They distribute long-lasting nets which don’t need retreating with insecticide. They are extremely transparent and monitor their own impact carefully, requiring photo verification from each net distribution. For more information, read GiveWell's detailed review, or the more accessible Charity Science summary.

How to donate

To find out which charities are tax-deductible in your country and get links to give to them tax-efficiently, you can use this interactive tool that I made. If you give this season, consider sharing the charities you choose on the EA Donation Registry. We can see which charities EAs pick, and which of the new ones prove popular!

I've heard of the concept of "weirdness points" many times before, but after a bit of searching I can't find a definitive post describing the concept, so I've decided to make one.  As a disclaimer, I don't think the evidence backing this post is all that strong and I am skeptical, but I do think it's strong enough to be worth considering, and I'm probably going to make some minor life changes based on it.

-

Chances are that if you're reading this post, you're probably a bit weird in some way.

No offense, of course.  In fact, I actually mean it as a compliment.  Weirdness is incredibly important.  If people weren't willing to deviate from society and hold weird beliefs, we wouldn't have had the important social movements that ended slavery and pushed back against racism, that created democracy, that expanded social roles for women, and that made the world a better place in numerous other ways.

Many things we take for granted now as why our current society as great were once... weird.

Joseph Overton theorized that policy develops through six stages: unthinkable, then radical, then acceptable, then sensible, then popular, then actual policy.  We could see this happen with many policies -- currently same-sex marriage is making its way from popular to actual policy, but not to long ago it was merely acceptable, and not too long before that it was pretty radical.

Some good ideas are currently in the radical range.  Effective altruism itself is such a collection of beliefs typical people would consider pretty radical.  Many people think donating 3% of their income is a lot, let alone the 10% demand that Giving What We Can places, or the 50%+ that some people in the community do.

And that's not all.  Others would suggest that everyone become vegetarian, advocating for open borders and/or universal basic income, theabolishment of gendered language, having more resources into mitigating existential risk, focusing on research into Friendly AI, cryonicsand curing death, etc.

While many of these ideas might make the world a better place if made into policy, all of these ideas are pretty weird.

Weirdness, of course, is a drawback.  People take weird opinions less seriously.

The absurdity heuristic is a real bias that people -- even you -- have.  If an idea sounds weird to you, you're less likely to try and believe it,even if there's overwhelming evidence.  And social proof matters -- if less people believe something, people will be less likely to believe it.  Lastly, don't forget the halo effect -- if one part of you seems weird, the rest of you will seem weird too!

(Update: apparently this concept is, itself, already known to social psychology as idiosyncrasy credits.  Thanks, Mr. Commenter!)

...But we can use this knowledge to our advantage.  The halo effect can work in reverse -- if we're normal in many ways, our weird beliefs will seem more normal too.  If we have a notion of weirdness as a kind of currency that we have a limited supply of, we can spend it wisely, without looking like a crank.

All of this leads to the following actionable principles:

Recognize you only have a few "weirdness points" to spend.  Trying to convince all your friends to donate 50% of their income to MIRI, become a vegan, get a cryonics plan, and demand open borders will be met with a lot of resistance.   But -- I hypothesize -- that if you pick one of these ideas and push it, you'll have a lot more success.

Spend your weirdness points effectively.  Perhaps it's really important that people advocate for open borders.  But, perhaps, getting people to donate to developing world health would overall do more good.  In that case, I'd focus on moving donations to the developing world and leave open borders alone, even though it is really important.  You should triage your weirdness effectively the same way you would triage your donations.

Clean up and look good.  Lookism is a problem in society, and I wish people could look "weird" and still be socially acceptable.  But if you're a guy wearing a dress in public, or some punk rocker vegan advocate, recognize that you're spending your weirdness points fighting lookism, which means less weirdness points to spend promoting veganism or something else.

Advocate for more "normal" policies that are almost as good.   Of course, allocating your "weirdness points" on a few issues doesn't mean you have to stop advocating for other important issues -- just consider being less weird about it.  Perhaps universal basic income truly would be a very effective policy to help the poor in the United States.  But reforming the earned income tax credit and relaxing zoning laws would also both do a lot to help the poor in the US, and such suggestions aren't weird.

Use the foot-in-door technique and the door-in-face technique.  The foot-in-door technique involves starting with a small ask and gradually building up the ask, such as suggesting people donate a little bit effectively, and then gradually get them to take the Giving What We Can Pledge.  The door-in-face technique involves making a big ask (e.g., join Giving What We Can) and then substituting it for a smaller ask, like the Life You Can Save pledge or Try Out Giving.

Reconsider effective altruism's clustering of beliefs.  Right now, effective altruism is associated strongly with donating a lot of money and donating effectively, less strongly with impact in career choice, veganism, and existential risk.  Of course, I'm not saying that we should drop some of these memes completely.  But maybe EA should disconnect a bit more and compartmentalize -- for example, leaving AI risk to MIRI, for example, and not talk about it much, say, on 80,000 Hours.  And maybe instead of asking people to both give more AND give more effectively, we could focus more exclusively on asking people to donate what they already do more effectively.

Evaluate the above with more research.  While I think the evidence base behind this is decent, it's not great and I haven't spent that much time developing it.  I think we should look into this more with a review of the relevant literature and some careful, targeted, market research on the individual beliefs within effective altruism (how weird are they?) and how they should be connected or left disconnected.  Maybe this has already been done some?

-

Also discussed on the EA Forum and EA Facebook group.

A belief is an acceptance that a statement is true or that something exists.   As aspiring rationalists, we strive for our beliefs to be true, accurate, and minimally biased.     

You seldom see a single belief floating around.  Typically beliefs tend to group into clusters and chains.  In other words, if I believe that I am turning my thoughts into written words right now, that is not an isolated belief.  My belief chain might look something like this:

I have sight ->  The image coming into my eyes is of something that is metallic with bright lights and little boxes -> It is similar to things that have been called “computers” before -> I am wiggling my fingers to make patterns ->  this is called typing -> I am typing on a computer -> the words I am thinking are being translated into writing.    

Why does it matter whether I see my beliefs as chains or whether I simply look at the highest level belief such as “the words I am thinking are being translated into written word”?

It matters because at each link in the chain of belief, there is potential for falsehood to be introduced.  The further I am away from the source of my high-level belief, the less likely my high-level belief is to be accurate.   

Say for example that a three year old is typing on their toy computer that does not have the standard typing functionality of my computer.  They could still have the same logic chain that I used:

I have sight ->  The image coming into my eyes is of something that is metallic with bright lights and little boxes -> It is similar to things that have been called “computers”  before -> I am wiggling my fingers to make patterns ->  this is  called typing -> I am typing on a computer -> the words I am  thinking are being translated into writing.    

Belief chains can be corrupted in many ways.  Here are a few:

1.   Our intuitions tell us that the more interconnecting beliefs we have, and the more agreement between different beliefs, the more likely they are to be true, right?  We can check them against each other and use them as confirming evidence for one another.

These interconnections can come from the beliefs we have accumulated in our own minds, and also from trust relationships with other people.  We use interconnecting beliefs from other people just as we use interconnecting beliefs in our own minds.  While not good or bad in and of itself, the down side of this system of validation is how we fall victim to the various types of groupthink.  

This is easiest to talk about with a diagram.  In these diagrams, we are assuming that truth (yellow T circles) comes from a source at the bottom of the diagram. Beliefs not originating from truth are labeled with a (B).   As aspiring rationalists, truth is what we want.   

What is truth?

Truth is a description reflecting the underlying fundamental structure of reality. The reality does not change regardless of what perspective you are looking at it from. As an example, "I think therefore I am" is something most people agree is obviously a truth. Most people agree that the laws of physics, in some version, are truths.

What is a source of truth?

A source of truth is the bottom level of stuff that composes whatever you're talking about.  If you're programming, the data you're manipulating breaks down into binary 0s and 1s.  But in order to let you handle it faster and more intuitively, it's assembled into layers upon layers of abstracted superstructures, until you're typing nearly English-like code into a preexisting program, or drawing a digital picture with a tablet pen in a very analog-feeling way.  Working directly with the source all the time isn't a good idea - in fact, it's usually unfeasible - and most problems with a higher-level abstraction shouldn't be patched by going all the way down.  But if you utterly disconnect from the fact that computers are in binary under their GUIs, or that no compass and paper can create a genuinely equation-perfect circle, or that physics isn't genuinely Newtonian under the hood - you'll have nowhere to backtrack to if it turns out there was a wrong turn in your reasoning.  You won't be able to sanity-check if you tell yourself a long twisty story about human motivations and "shoulds" and then come up with an action to take on that basis.

Below is a diagram of a healthy chain of pure true belief originating from a source of truth.  

2.   Belief chains can get disconnected from the source of truth.   For example, say that there is a group which has based their philosophy on the understanding of a certain physicist.   Say that the physicist dies, and that the group continues with expanding on that same belief set, although they have not yet integrated one of the key links that the physicist had which connected the chain to a source of truth.  In this case, you can end up with a cluster of belief that looks something like this:

You now have a cluster of belief, that contains some truth, but is no longer linked to source of truth, and fills in the gaps with ungrounded propositions.  This is the sort of situation that leads to high levels of overconfidence, and what Alexander Pope referred to when he wrote:  “A little learning is a dangerous thing."

What does this metaphor look like in real world terms?

Author's note: this post was written on Sunday, Oct. 19th. Its sequel will be written on Sunday, Oct. 27th.

Last night, I went to bed content with a fun and eventful weekend gone by. This morning, I woke up, took a shower, did my morning exercises, and began eat breakfast before making the commute up to work.

At the breakfast table, though, I was surprised to learn that it was Sunday, not Monday. I had misremembered what day it was and in fact had an entire day ahead of me with nothing on the agenda. At first, this wasn't very interesting, but then I started thinking. What to do with an entirely free day, without any real routine?

I realized that I didn't particularly know what to do, so I decided that I would simply live a day without defaults. At each moment of the day, I would act only in accordance with my curiosity and genuine interest. If I noticed myself becoming bored, disinterested, or otherwise less than enthused about what was going on, I would stop doing it.

What I found was quite surprising. I spent much less time doing routine activities like reading the news and browsing discussion boards, and much more time doing things that I've "always wanted to get around to"-- meditation, trying out a new exercise routine, even just spending some time walking around outside and relaxing in the sun.

Further, this seemed to actually make me more productive. When I sat down to get some work done, it was because I was legitimately interested in finishing my work and curious as to whether I could use a new method I had thought up in order to solve it. I was able to resolve something that's been annoying me for a while in much less time than I thought it would take.

By the end of the day, I started thinking "is there any reason that I don't spend every day like this?" As far as I can tell, there isn't really. I do have a few work tasks that I consider relatively uninteresting, but there are multiple solutions to that problem that I suspect I can implement relatively easily.

My plan is to spend the next week doing the same thing that I did today and then report back. I'm excited to let you all know what I find!

Recently, I found myself in a conversation with someone advocating the use of Knightian uncertainty. He (who I'm anonymizing as Sir Percy) made suggestions that are useful to most bounded reasoners, and which can be integrated into a Bayesian framework. He also claimed preferences that depend upon his Knightian uncertainty and that he's not an expected utility maximizer. Further, he claimed that Bayesian reasoning cannot capture his preferences. Specifically, Sir Percy said he maximizes minimum expected utility given his Knightian uncertainty, using what I will refer to as the "MMEU rule" to make decisions.

In my previous post, I showed that Bayesian expected utility maximizers can exhibit behavior in accordance with his preferences. Two such reasoners, Paranoid Perry and Cautious Caul, were explored. These hypothetical agents demonstrate that it is possible for Bayesians to be "ambiguity averse", e.g. to avoid certain types of uncertainty.

But Perry and Caul are unnatural agents using strange priors. Is this because we are twisting the Bayesian framework to represent behavior it is ill-suited to emulate? Or does the strangeness of Perry and Caul merely reveal a strangeness in the MMEU rule?

In this post, I'll argue the latter: maximization of minimum expected utility is not a good decision rule, for the same reason that Perry and Caul seem irrational. My rejection of the MMEU rule will follow from my rejections of Perry and Caul.

Since the episode with Eugine_Nier, I have received three private messages from different people asking me to investigate various cases of suspected mass downvoting. And to be quite honest, I don't want to deal with this. Eugine's case was relatively clear-cut, since he had engaged in systematic downvoting of a massive scale, but the new situations are a lot fuzzier and I'm not sure of what exactly the rules should be (what counts as a permitted use of the downvote system and what doesn't?).

At least one person has also privately contacted me and offered to carry out moderator duties if I don't want them, but even if I told them yes (on what basis? why them and not someone else?), I don't know what kind of policy I should tell them to enforce. I only happened to be appointed a moderator because I was in the list of top 10 posters at a particular time, and I don't feel like I should have any particular authority to make the rules. Nor do I feel like I have any good idea of what the rules should be, or who would be the right person to enforce them.

In any case, I don't want to be doing this job, nor do I particularly feel like being responsible for figuring out who should, or how, or what the heck. I've already started visiting LW less often because I dread having new investigation requests to deal with. So if you folks could be so kind as to figure it out without my involvement? If there's a clear consensus that someone in particular should deal with this, I can give them mod powers, or something.