Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.
The problem of published research findings not being reliable has been discussed here before.
One problem with RCTs that has received little attention is that, due to informed consent laws and ethical considerations, subjects are aware that they might be receiving sham therapy. This differs from the environment outside of the research setting, where people are confident that whatever their doctor prescribes is what they will get from their pharmacist. I can imagine many ways in which subjects' uncertainty about treatment assignment could affect outcomes (adherence is one possible mechanism). I wrote a short paper about this, focusing out what we would ideally estimate if we could lie to subjects, versus what we actually can estimate in RCTs (link). Here is the abstract:
It is widely recognized that traditional randomized controlled trials (RCTs) have limited generalizability due to the numerous ways in which conditions of RCTs differ from those experienced each day by patients and physicians. As a result, there has been a recent push towards pragmatic trials that better mimic real-world conditions. One way in which RCTs differ from normal everyday experience is that all patients in the trial have uncertainty about what treatment they were assigned. Outside of the RCT setting, if a patient is prescribed a drug then there is no reason for them to wonder if it is a placebo. Uncertainty about treatment assignment could affect both treatment and placebo response. We use a potential outcomes approach to define relevant causal effects based on combinations of treatment assignment and belief about treatment assignment. We show that traditional RCTs are designed to estimate a quantity that is typically not of primary interest. We propose a new study design that has the potential to provide information about a wider range of interesting causal effects
Any thoughts on this? Is this a trivial technical issue or something worth addressing?
View more: Next