Pragmatic Trials Collaborative

Measuring What Matters

Overestimation of benefit when clinical trials stop early: a simulation study

Summary

When we predicate our attention, or action, on a clinical trial demonstrating statistically significant benefit, we set ourselves up, on average, to overestimate that benefit. This is because trials that deviate towards greater benefit than is reality are more likely to meet our significance threshold – creating a selection bias that has been referred to as the "winner’s curse".1 In this study we simulate cardiovascular risk reduction trials typical of those being stopped early for benefit at interim analysis, and compare the apparent benefit when significance was reached, with the real benefit used to generate the data.

Bottom Line

Trialists exploring interventions targeted at cardiovascular risk reduction should reconsider stopping early if substantially fewer than 250 primary outcomes have been observed. Consumers of such trials should consider the possibility that estimations of benefit are meaningfully inflated.

Conclusion: Simulated trials typical of those stopping early for benefit overestimate the true relative risk reduction (RRR) by roughly 50% (i.e. the true RRR was 2/3 of the observed value). Overestimation was much smaller, and likely unimportant, when simulating large trials stopping early. Whether trials were large or small, stopped early or not, a minimum 250 events were needed to avoid overestimating relative risk reduction by an absolute 5% or more.

References
  1. Ioannidis JP. Why most discovered true associations are inflated. Epidemiology. 2008;19(5):640-8