Here are two more profiles of Bueno de Mesquita:
The NYT article provides the most detail I've seen on how Bueno de Mesquita's model actually works. Tetlock is wary of him for being a hedgehog, and concludes Bueno de Mesquita is better than other experts, but that's not saying much.
From an interview, here is Bueno de Mesquita's own recommendations on where to learn more: "Interested readers should read my 2002 book Predicting Politics (Ohio State University Press) or my 1994 book (co-edited with Frans Stokman) European Community Decision Making (Yale University Press) or my 1997 article in International Interactions for explanations of how the model works. They should also read Stanley Feder’s 2002 article in the Annual Review of Political Science for an evaluation of what the model can and cannot do based on the experiences of someone who used it more than a thousand times at the CIA."
Good links. The NYT article has criticism, as well as information about failed predictions:
While Bueno de Mesquita has published many predictions in academic journals, the vast majority of his forecasts have been done in secret for corporate or government clients, where no independent academics can verify them. “We have no idea if he’s right 9 times out of 10, or 9 times out of a hundred, or 9 times out of a thousand,” Walt says. Walt also isn’t impressed by Stanley Feder’s C.I.A. study showing Bueno de Mesquita’s 90 percent hit rate. “It’s one midlevel C.I.A. bureaucrat saying, ‘This was a useful tool,’ ” Walt says. “It’s not like he’s got Brent Scowcroft saying, ‘Back in the Bush administration, we didn’t make a decision without consulting Bueno de Mesquita.’ ” Other academics point out that rational-actor theory has come under increasing criticism in recent years, as more evidence accumulates that people make many decisions irrationally.
And it’s true that there have been cases when Bueno de Mesquita’s model has gone awry. In his 1996 book, “Red Flag Over Hong Kong,” he predicted that the press in Hong Kong “will become largely a tool of the state” — a highly debatable claim today. (In 2006, Reporters Without Borders noted concerns about self-censorship but said that “journalists remain free in Hong Kong.”) In early 1993, a corporate client asked him to forecast whether the Clinton administration’s health care plan would pass, and he said it would.
Your link to the NYT was hijacked, it now goes to a "anti-viruses" site - I closed the tab as quickly as I could and didn't catch much information about the site. Rather, the NYT site was apparently hijacked - I got redirected.
Works for me, right after you added this comment. The URL is fine. Perhaps you have been hacked or infected with some malware that is responsible for redirecting your browser. The URL is http://www.nytimes.com/2009/08/16/magazine/16Bruce-t.html?pagewanted=all, which is definitely a New York Times URL.
I actually got the virus redirect thing on a different NY Times link earlier today and I'm running Firefox 3 on Ubuntu.
I'm running Firefox 3.5; but the anti-viruses redirect is gone now, I just got through and read the article. I deleted my original comment, no sense in cluttering the thread unnecessarily, so I don't know if this is going to show up or not. anonym's responding comment disappeared from my browser. EDIT: Now it has reappeared along with Kevin's and this new comment (I hate software).
This is a bizarre claim:
He foresaw that China would reclaim Hong Kong 12 years before it happened
It's might be just a typo, but I penalize BdM for it. Britain and China negotiated the hand-over 13 years ahead, so 12 year predictions are unimpressive. Wikipedia dates the negotiation to the governor's 1979 trip to the mainland. (This comment seems to acknowledge but deny that version of history and instead put the focus on 15 year mortgages, which does match the 1982 beginning of serious negotiations. Yet earlier citizenship bills suggest that UK was planning for the handover.)
Do we have any serious hard-to-fake evidence that this is true? Examples given might all well be cherry-picked datapoints.
hard-to-fake evidence
The CIA report, which I haven't read, is pretty hard to fake. But it may not be so useful if we don't know how good is the baseline of CIA analysts and the questions. Tetlock says that it's very easy to write very simple computer programs that beat all of the people he assessed.
I read somewhere that both the CIA analysts and Bueno de Mesquita got 90%, whatever that means, but that he was more precise. I don't know how well-calibrated 90% is, but I suspect that the CIA analysts had more humility than any of Tetlock's experts, perhaps because they considered themselves cogs, rather than celebrities.
I assume he really did publish lots of predictions. But I wouldn't be shocked if he's blatantly lying and they're just wrong. Here's an example of someone contradicted about his predictions by his publications. In 1987, Michael Lewis begins his epilogue to Liar's Poker with "I didn't think the firm was doomed. I didn't think that Wall Street would collapse." Today, he writes "In the two decades since then, I had been waiting for the end of Wall Street." These quotes are not entirely fair, being cherry-picked from the midst of qualifications and explanations. But the similar language is striking.
(When I invoke Tetlock, I'm talking about his book. I don't know what he wrote about BdM.)
Tetlock says that it's very easy to write very simple computer programs that beat all of the people he assessed.
This is probably related to the fallacy of college admissions. Everyone in admissions thinks they can do better a job of predicting college success by using more criteria and their own professional judgement, and in every test using the SAT alone does better in aggregate.
For more on this and related phenomena, see Michael Bishop, "In Praise of Epistemic Irresponsibility: How Lazy And Ignorant Can You Be?"
in every test using the SAT alone does better in aggregate.
This is emphatically wrong. From the most pro-SAT source I can think of, SAT's + High School GPA are the most accurate predictor of first-year grades available. I admit the possibility that if you use a different measure of college success, my point is not valid.
It may be comforting to overstate the value of standardized testing, but that does not make it true, or excuse unfounded generalizations. One should be particularly careful to avoid such generalizations in a blue-green issue like the value of intelligence testing.
I think billswift's point was that the SAT is a better predictor as opposed to using human judgement as in reading the application letter, etc... It seems plausible that adding another "objective" measure like the HSGPA will further increase accuracy. But the big point is that using human judgement will DECREASE accuracy. The really interesting paper is the following as suggested by Alicorn:
Michael Bishop, "In Praise of Epistemic Irresponsibility: How Lazy And Ignorant Can You Be?"
I don't remember my sources, but I came across this in several published books in the 1990s (I think they were from the late 1980s and early 90s but I'm not sure). This was before they weakened the SAT, so maybe that accounts for the difference, or maybe you're going by ad copy, rather than cognitive psych reports, which are attempting to show the value of the tests without offending the consumers of the tests.
In one episode of EconTalk he said he had published a prediction of who Khomeini's successor would be years before the fact.
If the details in Good Magazine are correct, then it looks like costly signaling, not cherry-picking.
The details are that he was condemned at the APSA meeting so harshly that he got a public apology five years later when his prediction turned out to be correct. If he were making tons of controversial predictions and getting publicly condemned every meeting, the Iran expert should have felt no need to apologize when he's right once. It's important for this story that APSA is all of political science, not just Iran.
But these details look hard to check. I'm inclined to throw them out and call it cherry-picking.
I was hoping somebody would be familiar with his work from before. I didn't go hunting around for his papers, since I'm no expert on international politics and would have to spend as much if not more time figuring out what the outcomes were than researching his actual predictions.
Is he doing some of the best work in the area? Perhaps. Is he as good as some say? Definitely not.
As he sometimes acknowledges, the information he feeds into his models is incredibly important. He gets that information by interviewing experts in the field. One might be able to do better than any individual expert by merely averaging the opinions of several experts.
Good hour long interviews with him here: http://www.econtalk.org/archives/_featuring/bruce_bueno_de_mesquita/
You certainly can often do better than individual experts by averaging experts, but BDM's methods actually do add something.
To the extent that BDM predicts things like the Khomeini succession (and something about the Indian parliament, if I recall one of my political economy classes) in ways nobody else sees, including himself, it's worth understanding just how game theory allows unexpected or unintuitive results from completely basic inputs.
BDM's use of experts is confined to the inputs--a place where their judgments are more straightforward. It is rather easier to compare the relative power of two parliamentarians, for instance, than to predict the outcome of a multiparty election. Averaging experts' opinions on the latter may not improve forecasts much, and that's where BDM's innovations have been useful.
I should note too that he sometimes derives the model inputs himself.
I don't want to oversell his abilities, but he's doing pretty innovative work at the cutting edge of political economic theory.
To clarify the last point, I particularly recommend examining the selectorate theory in The Logic of Political Survival, chapters 2 and 3, and "Political Survival and Endogenous Institutional Change" in Comparative Political Studies 2008.
Both are from a recent course I took on political economy.
Here are a couple of things that annoy me. I think he's misleading about a lot of aspects of his methods, yet I don't doubt his his claims about his record, though I think we're missing a lot of context to assess it.
The India example in the NYT implies that he's using the computer to solve the NP problem of finding good coalitions. But once he has the coalition in hand, he should be able to take it to other people--that's the point of NP--and convince them that it's better than the one they suggested, yet this is exactly what he doesn't do in the India example. In fact, he claims not to understand the output himself! This claim about his methods also doesn't seem to match the dynamic point of view, as in his predictions about the path that Iranian politicians will travel before they reach equilibrium. (Part of that is that some of the goals are spread over time, like a certain amount of international attention. But he also implies at other times that he's predicting not just the best coalition, but how people will go about finding it. This is absurd.)
I liked this podcast about the work that Stephen Walt praises in NYT. I particularly liked the examples of Leopold II of Belgium and Chiang Kai-Shek as examples of people who ruled two countries each, with different results; and in Leopold's case, he ruled the two simultaneously. Sometimes BdM implies that this work is related to his predictions, but it doesn't seem so to me from other things he says. In particular, in his academic work, he seems to discount diversity of goals, while when he talks about Iran, he seems to claim that he's taking into account diversity of how much the actors care about the national glory of having a bomb, etc., rather than just caring about its reflection on themselves.
Well, in 21 months we'll be able to have some partial track record:
Those predictions are terribly vague. He produces an illusion of precise predictions about the timing of the power of various factions, as in graphs in his TED talk, but gives no explanation of what these numbers mean. It's not even clear who are the "moneyed interests." It's probably clear who are the different factions of mullahs, but how can we verify that one group has gained power? All this leaves of his prediction is that of the three choices of nothing nuclear, civilian power, and publicly endorsing weapons, he chooses the middle path, the status quo. (there's also the part about weapons grade uranium. I guess that's something.) Maybe always predicting the status quo gets you a better track record than most experts, but it's not a good enough track record to care.
The fact that he does marketing, is not evidence against his prediction abilities, but it is important not to be distracted by the marketing.
I stumbled upon an article called The New Nostradamus, reporting of a game-theoretic model which predicts political outcomes with startling effectiveness. The results are very impressive. However, the site hosting the article is unfamiliar to me, so I'm not certain of the article's verity, but a quick Google seems to support the claims, at least on a superficial skimming. Here's his TED talk. The model seems almost too good to be true, though. Anybody know more?
Some choice bits from the article:
The claim:
In fact, the professor says that a computer model he built and has perfected over the last 25 years can predict the outcome of virtually any international conflict, provided the basic input is accurate. What’s more, his predictions are alarmingly specific. His fans include at least one current presidential hopeful, a gaggle of Fortune 500 companies, the CIA, and the Department of Defense.
The results:
The criticism rankles him, because, to his mind, the proof is right there on the page. “I’ve published a lot of forecasting papers over the years,” he says. “Papers that are about things that had not yet happened when the paper was published but would happen within some reasonable amount of time. There’s a track record that I can point to.” And indeed there is. Bueno de Mesquita has made a slew of uncannily accurate predictions—more than 2,000, on subjects ranging from the terrorist threat to America to the peace process in Northern Ireland—that would seem to prove him right.
[...]
To verify the accuracy of his model, the CIA set up a kind of forecasting face-off that pit predictions from his model against those of Langley’s more traditional in-house intelligence analysts and area specialists. “We tested Bueno de Mesquita’s model on scores of issues that were conducted in real time—that is, the forecasts were made before the events actually happened,” says Stanley Feder, a former high-level CIA analyst. “We found the model to be accurate 90 percent of the time,” he wrote. Another study evaluating Bueno de Mesquita’s real-time forecasts of 21 policy decisions in the European community concluded that “the probability that the predicted outcome was what indeed occurred was an astounding 97 percent.” What’s more, Bueno de Mesquita’s forecasts were much more detailed than those of the more traditional analysts. “The real issue is the specificity of the accuracy,” says Feder. “We found that DI (Directorate of National Intelligence) analyses, even when they were right, were vague compared to the model’s forecasts. To use an archery metaphor, if you hit the target, that’s great. But if you hit the bull’s eye—that’s amazing."
Gets good money for it:
Though controversial in the academic world, Bueno de Mesquita and his model have proven quite popular in the private sector. In addition to his teaching responsibilities and consulting for the government, he also runs a successful private business, Mesquita & Roundell, with offices in Rockefeller Center. Advising some of the top companies in the country, he earns a tidy sum: Mesquita & Roundell’s minimum fee is $50,000 for a project that includes two issues. Most projects involve multiple issues. “I’m not selling my wisdom,” he says. “I’m selling a tool that can help them get better results. That tool is the model.”
“In the private sector, we deal with three areas: litigation, mergers and acquisitions, and regulation,” he says. “On average in litigation, we produce a settlement that is 40 percent better than what the attorneys think is the best that can be achieved.” While Bueno de Mesquita’s present client list is confidential, past clients include Union Carbide, which needed a little help in structuring its defense after its 1984 chemical-plant disaster in Bhopal, India, claimed the lives of an estimated 22,000 people; the giant accounting firm Arthur Andersen; and British Aerospace during its merger with GEC-Marconi.
The method should be of special interest to the OB/LW audience, as it brings to mind discussions about self-deception and evolutionary vs. acknowledged goals and behavior:
Which illustrates the next incontrovertible fact about game theory: In the foreboding world view of rational choice, everyone is a raging dirtbag. Bueno de Mesquita points to dictatorships to prove his point: “If you liberate people from the constraint of having to satisfy other people in order to advance themselves, people don’t do good things.” When analyzing a problem in international relations, Bueno de Mesquita doesn’t give a whit about the local culture, history, economy, or any of the other considerations that more traditional political scientists weigh. In fact, rational choicers like Bueno de Mesquita tend to view such traditional approaches with a condescension bordering on disdain. “One is the study of politics as an expression of personal opinion as opposed to political science,” he says dryly. His only concern is with what the political actors want, what they say they want (often two very different things), and how each of their various options will affect their career advancement. He feeds this data into his computer model and out pop the answers.
[...]
In his continuing work for the CIA and the Defense Department, one of his most recent assignments has been North Korea and its nuclear program. His analysis starts from the premise that what Kim Jong Il cares most about is his political survival. As Bueno de Mesquita sees it, the principal reason for his nuclear program is to deter the United States from taking him out, by raising the costs of doing so. “The solution, then, lies in a mechanism that guarantees us that he not use these weapons and guarantees him that we not interfere with his political survival,” he says.
[...]
Recently, he’s applied his science to come up with some novel ideas on how to resolve the Israeli-Palestinian conflict. “In my view, it is a mistake to look for strategies that build mutual trust because it ain’t going to happen. Neither side has any reason to trust the other, for good reason,” he says. “Land for peace is an inherently flawed concept because it has a fundamental commitment problem. If I give you land on your promise of peace in the future, after you have the land, as the Israelis well know, it is very costly to take it back if you renege. You have an incentive to say, ‘You made a good step, it’s a gesture in the right direction, but I thought you were giving me more than this. I can’t give you peace just for this, it’s not enough.’ Conversely, if we have peace for land—you disarm, put down your weapons, and get rid of the threats to me and I will then give you the land—the reverse is true: I have no commitment to follow through. Once you’ve laid down your weapons, you have no threat.”
Bueno de Mesquita’s answer to this dilemma, which he discussed with the former Israeli prime minister and recently elected Labor leader Ehud Barak, is a formula that guarantees mutual incentives to cooperate. “In a peaceful world, what do the Palestinians anticipate will be their main source of economic viability? Tourism. This is what their own documents say. And, of course, the Israelis make a lot of money from tourism, and that revenue is very easy to track. As a starting point requiring no trust, no mutual cooperation, I would suggest that all tourist revenue be [divided by] a fixed formula based on the current population of the region, which is roughly 40 percent Palestinian, 60 percent Israeli. The money would go automatically to each side. Now, when there is violence, tourists don’t come. So the tourist revenue is automatically responsive to the level of violence on either side for both sides. You have an accounting firm that both sides agree to, you let the U.N. do it, whatever. It’s completely self-enforcing, it requires no cooperation except the initial agreement by the Israelis that they are going to turn this part of the revenue over, on a fixed formula based on population, to some international agency, and that’s that.”