Pragmatic Trials Collaborative
Measuring What Matters
Overestimation of benefit when clinical trials stop early: a simulation study
Summary
When we predicate our attention, or action, on a clinical trial demonstrating statistically significant benefit, we set ourselves up, on average, to overestimate that benefit. This is because trials that deviate towards greater benefit than is reality are more likely to meet our significance threshold – creating a selection bias that has been referred to as the "winner’s curse".1 In this study we simulate cardiovascular risk reduction trials typical of those being stopped early for benefit at interim analysis, and compare the apparent benefit when significance was reached, with the real benefit used to generate the data.
Conclusion: Simulated trials typical of those stopping early for benefit overestimate the true relative risk reduction (RRR) by roughly 50% (i.e. the true RRR was 2/3 of the observed value). Overestimation was much smaller, and likely unimportant, when simulating large trials stopping early. Whether trials were large or small, stopped early or not, a minimum 250 events were needed to avoid overestimating relative risk reduction by an absolute 5% or more.
Bottom Line
Trialists exploring interventions targeted at cardiovascular risk reduction should reconsider stopping early if substantially fewer than 250 primary outcomes have been observed. Consumers of such trials should consider the possibility that estimations of benefit are meaningfully inflated.