= 27d567bc2d48d9f40e9ccc20ea370b15)
4 Common Prediction Failures and How to Fix Them [LINK]
A post I wrote on CFAR's blog about the errors we make in hedonic prospecting, based on Gilbert and Wilson's 2007 review article of the topic. Let me know what you think! Reposting from the discussion in case anyone missed it. :)
= 783df68a0f980790206b9ea87794c5b6)
Failure Modes sometimes correspond to Game Mechanics
If you want to carry a brimming cup of coffee without spilling it, you may want to "change" your goal to instead primarily concentrate on humming. This is an example of a general pattern. It sometimes helps to focus on a nearby artificial goal rather than your actual goal. Let me call that strategy "gamification". There is a business strategy, also named "gamification", of adding game mechanics to a website in order to achieve various business goals. This is related but different. Here I'm referring to a strategy for problem solvers.
We sometimes fail, and sometimes one failure is very similar to another failure. That is, there are characteristic ways that we fail. One of the primary ways that we can improve is to learn our failure modes and create external structures (pieces of paper, software tools) that check, protect against, or head off those forms of failure.
For example, imagine this plan of checklist improvement:
- Change your normal way of working to include an explicit checklist (that starts empty).
- When you make a mistake:
- Analyze what went wrong
- Try to generalize the particular incident to a category
- Add an item to your checklist.
This is very simple and generic, but it is reasonable to believe that if you carefully and diligently followed this plan, your reliability would go up (with diminishing returns because your errors are also your opportunities for improvement). I have not read Mayo, but her "error-theoretic" philosophy of science might be applicable here.
We can try to build a correspondence between failure modes, and game mechanics that attempt to cope for that failure mode.
Dealing with the high quantity of scientific error in medicine
In a recent article, John Ioannidis describes a very high proportion of medical research as wrong.
Still, Ioannidis anticipated that the community might shrug off his findings: sure, a lot of dubious research makes it into journals, but we researchers and physicians know to ignore it and focus on the good stuff, so what’s the big deal? The other paper headed off that claim. He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.
Part of the problem is that surprising results get more interest, and surprising results are more likely to be wrong. (I'm not dead certain of this-- if the baseline beliefs are highly likely to be wrong, surprising beliefs become somewhat less likely to be wrong.) Replication is boring. Failure to replicate a bright shiny surprising belief is boring. A tremendous amount isn't checked, and that's before you start considering that a lot of medical research is funded by companies that want to sell something.
Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
The culture at LW shows a lot of reliance on small inferential psychological studies-- for example that doing a good deed leads to worse behavior later. Please watch out for that.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}