Cross-posted from Huffington Post. See also The End of Bullshit at the Hands of Critical Rationalism.

Debating season is in full swing, and as per usual the presidential candidates are playing fast and loose with the truth. Fact-checking sites such as PolitiFact and FactCheck.org have had plenty of easy targets in the debates so far. For instance, in the CNN Republican debate on September 16, Fiorina made several dubious claims about the Planned Parenthood video, as did Cruz about the Iran agreement. Similarly, in the CNN Democratic debate on October 13, Sanders falsely claimed that the U.S. has "more wealth and income inequality than any other country", whereas Chafee fudged the data on his Rhode Island record. No doubt we are going to see more of that in the rest of the presidential campaign. The fact-checkers won't need to worry about finding easy targets.

Research shows that fact-checking actually does make a difference. Incredible as it may seem, the candidates would probably have been even more careless with the truth if it weren't for the fact-checkers. To some extent, fact-checkers are a deterrent to politicians inclined to stretch the truth.

At the same time, the fact that falsehoods and misrepresentations of the truth are still so common shows that this deterrence effect is not particularly strong. This raises the question how we can make it stronger. Is there a way to improve on PolitiFact's and FactCheck.org's model - Fact-Checking 2.0, if you will?

Spencer Greenberg of ClearerThinking and I have developed a tool which we hope could play that role. Greenberg has created an application to embed videos of recorded debates and then add subtitles to them. In these subtitles, I point out falsehoods and misrepresentations of the truth at the moment when the candidates make them. For instance, when Fiorina says about the Planned Parenthood video that there is "a fully formed fetus on the table, its heart beating, its legs kicking, while someone says we have to keep it alive to harvest its brain", I write in the subtitles:

We think that reading that a candidate's statement is false just as it is made could have quite a striking effect. It could trigger more visceral feelings among the viewers than standard fact-checking, which is published in separate articles. To over and over again read in the subtitles that what you're being told simply isn't true should outrage anyone who finds truth-telling an important quality.

Another salient feature of our subtitles is that we go beyond standard fact-checking. There are many other ways of misleading the audience besides playing fast and loose with the truth, such as evasions, ad hominem-attacks and other logical fallacies. Many of these are hard to spot for the viewers. We must therefore go beyond fact-checking and also do argument-checking, as we call it. If fact-checking grew more effective, and misrepresenting the truth less viable a strategy, politicians presumably would more frequently resort to Plan B: evading questions where they don't want the readers to know the truth. To stop that, we need careful argument-checking in addition to fact-checking.

So far, I've annotated the entire CNN Republican Debate, a 12 minute video from the CNN Democratic Debate (more annotations of this debate will come) and nine short clips (1-3 minutes) from the Fox News Republican Debate (August 6). My aim is to be as complete as possible, and I think that I've captured an overwhelming majority of the factual errors, evasions, and fallacies in the clips. The videos can be found on ClearerThinking as well as below.

The CNN Republican debate, subtitled in full.



The first 12 minutes of the CNN Democratic debate.

Nine short clips from the Fox News Debate: Christie and Paul, Bush, Carson, Cruz, Huckabee, Kasich, Rubio, Trump, Walker.

What is perhaps most striking is the sheer number of falsehoods, evasions and fallacies the candidates make. The 2hr 55 min long CNN Republican debate contains 273 fact-checking and argument-checking comments (many of which refer to various fact-checking sites). In total, 27 % of the video is subtitled. Similar numbers hold for the other videos.

Conventional wisdom has it that politicians lie and deceive on a massive scale. My analyses prove conventional wisdom right. The candidates use all sorts of trickery to put themselves in a better light and smear their opponents.

All of this trickery is severely problematic from several perspectives. Firstly, it is likely to undermine the voters' confidence in the political system. This is especially true for voters on the losing side. Why be loyal to a government which has gained power by misleading the electorate? No doubt many voters do think in those terms, more or less explicitly.

It is also likely to damage the image of democracy. The American presidential election is followed all over the world by millions if not billions of people. Many of them live in countries where democracy activists are struggling to amass support against authoritarian regimes. It hardly helps them that the election debates in the U.S. and other democratic countries look like this.

All of these deceptive arguments and claims also make it harder for voters to make informed decisions. Televised debates are supposed to help voters to get a better view of the candidates' policies and track-records, but how could they, if they can't trust what is being said? This is perhaps the most serious consequence of poor debates, since it is likely to lead to poorer decisions on the part of the voters, which in turn will lead to poorer political leadership and poorer policies.

Besides functioning as a more effective lie deterrent to the candidates, improved fact-checking could also nudge the networks to adjust the set-up of the debates. The way the networks lead the debates today hardly encourages serious and rational argumentation. To the contrary, they often positively goad the candidates against each other. Improved fact-checking could make it more salient to the viewers how poor the debates are, and induce them to demand a better debate set-up. The networks need to come up with a format which incentivizes the candidates to argue fairly and truthfully, and which makes it clear who has not. For instance, they could broadcast the debate again the next day, with fact-checking and argument-checking subtitles.

Another means to improve the debates is further technological innovation. For example, there should be a video annotation equivalent to Genius.com, the web application which allows you to annotate text on any webpage in a convenient way. That would be very useful for fact-checking and argument-checking purposes.

Fact-checking could even become automatic, as Google CEO Eric Schmidt predicted it would be within five years in 2006. Though Schmidt was over-optimistic, Google algorithms are able to fact-check websites with a high degree of accuracy today, whilst Washington Post already has built a rudimentary automatic fact-checker.

But besides new software applications and better debating formats, we also need something else, namely a raised awareness among the public what a great problem politicians' careless attitude to the truth is. They should ask themselves: are people inclined to mislead the voters really suited to shape the future of the world?

Politicians are normally held to high moral standards. Voters tend to take very strict views on other forms of dishonest behavior, such as cheating and tax evasion. Why, then, is it that they don't take a stricter view on intellectual dishonesty? Besides being morally objectionable, intellectual dishonesty is likely to lead to poor decisions. Voters would therefore be wise to let intellectual honesty be an important criterion when they cast their vote. If they started doing that on a grand scale, that would do more to improve the level of political debate than anything else I can think of.

Thanks to Aislinn Pluta, Doug Moore, Janko Prester, Philip Thonemann, Stella Vallgårda and Staffan Holmberg for their contributions to the annotations.

The prediction expert Philip Tetlock writes in New York Times on the power of precise predictions to counter political polarization. Note the similarity to Robin Hanson's futarchy idea.

IS there a solution to this country’s polarized politics?

Consider the debate over the nuclear deal with Iran, which was one of the nastiest foreign policy fights in recent memory. There was apocalyptic rhetoric, multimillion-dollar lobbying on both sides and a near-party-line Senate vote. But in another respect, the dispute was hardly unique: Like all policy debates, it was, at its core, a contest between competing predictions.

Opponents of the deal predicted that the agreement would not prevent Iran from getting the bomb, would put Israel at greater risk and would further destabilize the region. The deal’s supporters forecast that it would stop (or at least delay) Iran from fielding a nuclear weapon, would increase security for the United States and Israel and would underscore American leadership.

The problem with such predictions is that it is difficult to square them with objective reality. Why? Because few of them are specific enough to be testable. Key terms are left vague and undefined. (What exactly does “underscore leadership” mean?) Hedge words like “might” or “could” are deployed freely. And forecasts frequently fail to include precise dates or time frames. Even the most emphatic declarations — like former Vice President Dick Cheney’s prediction that the deal “will lead to a nuclear-armed Iran” — can be too open-ended to disconfirm.

//

Non-falsifiable predictions thus undermine the quality of our discourse. They also impede our ability to improve policy, for if we can never judge whether a prediction is good or bad, we can never discern which ways of thinking about a problem are best.

The solution is straightforward: Replace vague forecasts with testable predictions. Will the International Atomic Energy Agency report in December that Iran has adequately resolved concerns about the potential military dimensions of its nuclear program? Will Iran export or dilute its quantities of low-enriched uranium in excess of 300 kilograms by the deal’s “implementation day” early next year? Within the next six months, will any disputes over I.A.E.A. access to Iranian sites be referred to the Joint Commission for resolution?

Such questions don’t precisely get at what we want to know — namely, will the deal make the United States and its allies safer? — but they are testable and relevant to the question of the Iranian threat. Most important, they introduce accountability into forecasting. And that, it turns out, can depolarize debate.

In recent years, Professor Tetlock and collaborators have observed this depolarizing effect when conducting forecasting “tournaments” designed to identify what separates good forecasters from the rest of us. In these tournaments, run at the behest of the Intelligence Advanced Research Projects Activity (which supports research relevant to intelligence agencies), thousands of forecasters competed to answer roughly 500 questions on various national security topics, from the movement of Syrian refugees to the stability of the eurozone.

The tournaments identified a small group of people, the top 2 percent, who generated forecasts that, when averaged, beat the average of the crowd by well over 50 percent in each of the tournament’s four years. How did they do it? Like the rest of us, these “superforecasters” have political views, often strong ones. But they learned to seriously consider the possibility that they might be wrong.

What made such learning possible was the presence of accountability in the tournament: Forecasters were able see their competitors’ predictions, and that transparency reduced overconfidence and the instinct to make bold, ideologically driven predictions. If you can’t hide behind weasel words like “could” or “might,” you start constructing your predictions carefully. This makes sense: Modest forecasts are more likely to be correct than bold ones — and no one wants to look stupid.

This suggests a way to improve real-world discussion. Suppose, during the next ideologically charged policy debate, that we held a public forecasting tournament in which representatives from both sides had to make concrete predictions. (We are currently sponsoring such a tournament on the Iran deal.) Based on what we have seen in previous tournaments, this exercise would decrease the distance between the two camps. And because it would be possible to determine a “winner,” it would help us learn whether the conservative or liberal assessment of the issue was more accurate.

 

Either way, we would begin to emerge from our dark age of political polarization.

On average, matching donations supposedly do increase charitable giving (though I want to see more rigorous research on this - tips are welcome). One criticism against them is, though, that they are "illusory" - that is, that the matching donor eventually donates the same amount whether smaller donors match their donations or not. That means that a dollar from a smaller donor doesn't actually cause the matcher to contribute more.

One way to make matching donations real, as opposed to illusory, is this. Suppose that the matching donor is indifferent between donating to two charities (e.g. Against Malaria Foundation and MIRI). At the same time, lots of small donors think that one of them is far better than the other. Also, suppose that the matching donor sets the terms so that it's virtually certain that their whole matching fund will be used up (this could be done, e.g. by making the matching ratio very favourable).

Under these conditions, it will make a difference whether a small donor contributes or not, since if you don't, chances are that your donation will be replaced by a donation to the other charity. That means that a dollar from you as a smaller donor on average does cause the matcher to contribute more to your favourite charity.

This suggest a more general strategy for leveraging charity contributions. You could set up a set of matching funds, to which small donors could contribute. These funds would be "disjunctive" - they would match contributions to, e.g. AMF or MIRI, Open Borders or MSF or The Humane League, etc. The funds would from time to time declare that they match any donations to their target charities, and supporters of the respective target charities would start competing, in effect, for the matching donations.

In the simplest system, only people who are more or less indifferent between the target charities would donate to the matching funds. A somewhat more complex system incentivizes people who prefer one of the target charities, A, to give to the matching fund. Under such a system, an "A-ear-marked" donation to the matching fund would increase the matching donations (e.g. from 1:1 to 3:2) to A, and decrease matching donations to the other target charities the matching fund supports. That will, in turn, incentivize more giving to A relative to the other target charities. It is important that such adjustments are done in the right way, though. If, e.g. supporters of A has contributed 70 % of the matching fund, and supporters of B 30 %, then roughly 70 % of the extra money the matching fund generates (thanks to additional donations) should go to A, and 30 % to B. (It could actually get even more complicated than that, but let us leave this thread here for now.)

If such a system of matching funds was set up, an important question would be: should you donate to a matching fund, or donate to a target charity, and get your donations matched by a matching fund? Suppose that you expect those running the matching funds to adjust the matching ratios so that any donation to them that is ear-marked for your favourite charity A means that all extra donations your donation generates will go to A. In other words, if each dollar to the matching fund generates X cents in extra donations, you giving an A-ear-marked donation will mean X more cents to A. Then your decision will depend on:*

1) The size of X.

2) Your opinion of the charities competing with A in various matching funds. The better you think they are, the less reason you have to donate directly to A (since then you care less about money not going to A).

3) Replaceability effects. If you don't donate to A, who will replace you? Someone donating to A, or to some other charity? The more likely you think it is that you will be replaced by another donor to A, the less reason you have to donate directly to A.

4) The matching fund's matching ratio Y.

Suppose, for instance, that X = .2, that you think that the competitors to A in a particular matching fund generate zero utility, and that the probability that your donation will be replaced by another A donor is 50 %. Then you should choose to contribute to the matching fund if Y < .4:1, and donate directly if Y > .4:1.

You could set up a whole stock exchange, where people could buy shares in matching funds, and make donations to charities that will be matched by matching funds. It's an interesting question what the average level of matching would be in such a system. The higher it would be, the more charitable giving it would presumably generate. Therefore, one should to increase that level beyond .4:1 (beyond which people will start donating to the target charity in our example), which is not very high. For instance, you could tweak the system in a way that incentivizes matching, or you could try to get large donors or even the government to exclusively give matching donations.

These complex issues are still a bit foggy to me, and I might have made some mistakes. Any comments are welcome. See also this text on the EA forum where a similar system involving the government as the matching donor is discussed. This is an instance of Moral Trade, a concept developed by Toby Ord.

* If you don't think that, your opinion of whether A will get more or less than X extra cents because of your donation is a fifth parameter to consider.

Cross-posted from the EA forum. I asked for questions for this test here on LW about a year ago. Thanks to those who contributed.

Rationally, your political values shouldn't affect your factual beliefs. Nevertheless, that often happens. Many factual issues are politically controversial - typically because the true answer makes a certain political course of action more plausible - and on those issues, many partisans tend to disregard politically uncomfortable evidence.

This sort of political bias has been demonstrated in a large number of psychological studies. For instance, Yale professor Dan Kahan and his collaborators showed in a fascinating experiment that on politically controversial questions, people are quite likely to commit mathematical mistakes that help them retain their beliefs, but much less likely to commit mistakes that would force them to give up those belies. Examples like this abound in the literature.

Political bias is likely to be a major cause of misguided policies in democracies (even the main one according to economist Bryan Caplan). If they don’t have any special reason not to, people without special knowledge defer to the scientific consensus on technical issues. Thus, they do not interfere with the experts, who normally get things right. On politically controversial issues, however, they often let their political bias win over science and evidence, which means they’ll end up with false beliefs. And, in a democracy voters having systematically false beliefs obviously more often than not translates into misguided policy.

Can we reduce this kind of political bias? I’m fairly hopeful. One reason for optimism is that debiasing generally seems to be possible to at least some extent. This optimism of mine was strengthened by participating in a CFAR workshop last year. Political bias seems not to be fundamentally different from other kinds of biases and should thus be reducible too. But obviously one could argue against this view of mine. I’m happy to discuss this issue further.

Another reason for optimism is that it seems that the level of political bias is actually lower today than it was historically. People are better at judging politically controversial issues in a detached, scientific way today than they were in, say, the 14th century. This shows that progress is possible. There seems to be no reason to believe it couldn’t continue.

A third reason for optimism is that there seems to be a strong norm against political bias. Few people are consciously and intentionally politically biased. Instead most people seem to believe themselves to be politically rational, and hold that as a very important value (or so I believe). They fail to see their own biases due to the bias blind spot (which disables us from seeing our own biases).

Thus if you could somehow make it salient to people that they are biased, they would actually want to change. And if others saw how biased they are, the incentives to debias would be even stronger.

There are many ways in which you could make political bias salient. For instance, you could meticulously go through political debaters’ arguments and point out fallacies, like I have done on my blog. I will post more about that later. Here I want to focus on another method, however, namely a political bias test which I have constructed with ClearerThinking, run by EA-member Spencer Greenberg. Since learning how the test works might make you answer a bit differently, I will not explain how the test works here, but instead refer either to the explanatory sections of the test, or to Jess Whittlestone’s (also an EA member) Vox.com-article.

Our hope is of course that people taking the test might start thinking more both about their own biases, and about the problem of political bias in general. We want this important topic to be discussed more. Our test is produced for the American market, but hopefully, it could work as a generic template for bias tests in other countries (akin to the Political Compass or Voting Advice Applications).

Here is a guide for making new bias tests (where the main criticisms of our test are also discussed). Also, we hope that the test could inspire academic psychologists and political scientists to construct full-blown scientific political bias tests.

This does not mean, however, that we think that such bias tests in themselves will get rid of the problem of political bias. We need to attack the problem of political bias from many other angles as well.

Follow-up to Reverse Engineering of Belief Structures

Pro-con-lists of arguments such as ProCon.org and BalancedPolitics.org fill a useful purpose. They give an overview over complex debates, and arguably foster nuance. My network for evidence-based policy is currently in the process of constructing a similar site in Swedish.

I'm thinking it might be interesting to add more features to such a site. You could let people create a profile on the site. Then you would let them fill in whether they agree or disagree with the theses under discussion (cannabis legalization, GM foods legalization, etc), and also whether they agree or disagree with the different argument for and against these theses (alternatively, you could let them rate the arguments from 1-5).

Once you have this data, you could use them to give people different kinds of statistics. The most straightforward statistic would be their degree of "onesidedness". If you think that all of the arguments for the theses you believe in are good, and all the arguments against them are bad, then you're defined as onesided. If you, on the other hand, believe that some of your own side's arguments are bad, whereas some of the opponents' arguments are good, you're defined as not being onesided. (The exact mathematical function you would choose could be discussed.)

Once you've told people how one-sided they are, according to the test, you would discuss what might explain onesidedness. My hunch is that the most plausible explanation normally is different kinds of bias. Instead of reviewing new arguments impartially, people treat arguments for their views more leniently than arguments against their views. Hence they end up being onesided, according to the test.

There are other possible explanations, though. One is that all of the arguments against the thesis in question actually are bad. That might happen occassionally, but I don't think that's very common. As Eliezer Yudkowsky says in "Policy Debates Should Not Appear One-sided":

On questions of simple fact (for example, whether Earthly life arose by natural selection) there's a legitimate expectation that the argument should be a one-sided battle; the facts themselves are either one way or another, and the so-called "balance of evidence" should reflect this.  Indeed, under the Bayesian definition of evidence, "strong evidence" is just that sort of evidence which we only expect to find on one side of an argument.

But there is no reason for complex actions with many consequences to exhibit this onesidedness property.  

Instead, the reason why people end up with one-sided beliefs is bias, Yudkowsky argues:

Why do people seem to want their policy debates to be one-sided?

Politics is the mind-killer.  Arguments are soldiers.  Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back.  If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary.

Especially if you're consistently one-sided in lots of different debates, it's hard to see that any other hypothesis besides bias is plausible. It depends a bit on what kinds of arguments you include in the list, though. In our lists we haven't really checked the quality of the arguments (our purpose is to summarize the debate, rather than to judge it), but you could also do that, of course.

My hope is that such a test would make people more aware both of their own biases, and of the problem of political bias in general. I'm thinking that is the first step towards debiasing. I've also constructed a political bias test with similar methods and purposes together with ClearerThinking, which should be released soon.

You could also add other features to a pro-con-list. For instance, you could classify arguments in different ways: ad hominem-arguments, consequentialist arguments, rights-based arguments, etc. (Some arguments might be hard to classify, and then you just wouldn't do that. You wouldn't necessarily have to classify every argument.) Using this info, you could give people a profile: e.g., what kinds of arguments do they find most persuasive? That could make them reflect more on what kinds of arguments really are valid.

You could also combine these two features. For instance, some people might accept ad hominem-arguments when they support their views, but not when they contradict them. That would make your use of ad hominem-arguments onesided.

Yet another feature that could be added is a standard political compass. Since people fill in what theses they believe in (cannabis legalization, GM goods legalization, etc) you could calcluate what party is closest to them, based on the parties' stances on these issues. That could potentially make the test more attractive to take.

Suggestions of more possible features are welcome, as well as general comments - especially about implementation.