Part 2: 'Which experts to trust', 'Limitations' and 'Practice'

**Part 1: IS EXPERT OPINION A WASTE OF TIME? is available here

which experts to trust

Now for an application:

Tetlock (2005) showed, based on 20 years of longitudinal research on several hundred political experts, the single most powerful predictor of forecasting skill had little to do with what experts think, and more to do with how they think: he called it ‘cognitive style’. As well as being a predictor of forecasting accuracy, ‘cognitive style’ is also a strong correlate of overconfidence. Cognitive style is determined in large part by personality traits, which he characterized as hedgehogs and foxes.

Hedgehogs know “one big thing, and under the banner of parsimony [work] to expand the explanatory power of that big thing to ‘cover’ new cases”; the foxes know “many little things and [are] content to improvise ad hoc solutions” (p. 20-21). Cognitive style correlates with personality measures such as‘openness’ and ‘need for closure’. Tetlock’s work showed consistently greater overconfidence in ‘hedgehog’ experts than in ‘fox’ experts.

An obvious example of a category of hedgehog that springs to mind are ideologues – everyone from Anarchocapitalists to Bayesians to ‘materialists’ to ‘ordinary people’ to ‘Agnostics’ (just trying my best to insult the most number of people here, cause not enough people recognise they’re as guilty of the things they might be looking for in others). The motivated reasoning bias springs to mind (after prompting by the ACERA paper).

Commiserate:

Demographic variables had little bearing on accuracy. Years of experience also failed to predict accuracy. Left vs Right (the Ideology factor), Institutional vs Realist (the Realpolitik factor) and Doomster vs Boomster (the Optimism factor) also failed to predict accuracy. The consistent result across the spectrum of content domains, was that hedgehogs were: a) less well calibrated, b) had poorer discrimination, c) were more overconfident and d) were less likely to update their beliefs (were poorer Bayesians). Foxes outperformed all groups on each of the four criteria, and the middling groups fell in between in predictable ways.

important limitations to this research line

However, there have been few, if any, studies of their efficacy in realistic biosecurity settings. In particular, relatively little attention has been paid to the reliability or accuracy of conceptual models. Few of the formal techniques for elicitation, calibration or verification have been tested in conditions typical of biosecurity risk analysis. There is an opportunity to evaluate the most promising methods, with a view to implementing some of the most effective procedures in routine risk analyses.

Practice

For any smart cookies out there, you must wonder – well, what can you do to get the most out of experts? It’s too much information to tell you about here, but I recommend ACERA’s

• Elicitation tool, user manual and

• Elicitation tool, process manual

Academic delphi style groups outperform baseline groups by around 50%. Could professional delphi groups be formed to profit from stock and prediction markets?

and

• UPenn’s Delphi decision aid And this Real time Delphi tool

Sincerely, Carlos Larity

Self-appointed claimant as the resident expert expert

*Penned in order to get my karma back at around 11. I aimed to have a net zero karma – balancing more controversial stuff with popular content, just enough to not discredit myself. Though, I just found out that I actually can’t upvote when my karma is zero, and I need ’11 more’ to do so.

And, just a fun additional link on reddit for you to check out. Remember not to trust experts, particularly not experts in experts...or hedgehogs!