Pragmatic Trials Collaborative

Measuring What Matters

What is so important about pragmatic trials?*

Although randomized controlled trials are the cornerstone of medical evidence, and the basis for all treatment guidelines, there are systematic gaps in this evidence which need to be filled.

What is a pragmatic trial?1

Most randomized controlled trials (RCTs) are "explanatory" or "efficacy" trials, designed to answer whether a treatment can demonstrate benefit in a select population. Such trials normally compare an intervention to placebo or to another active intervention that may not be standard of care. They also focus on a narrow set of outcomes to which the intervention is targeted. In contrast, "pragmatic" or "effectiveness" trials: 1) examine interventions in a broader population representative of those who will be treated; 2) report outcomes of importance to patients that are intended to capture global benefit and harm; and 3) often compare the intervention in question to standard therapies. Compared to efficacy trials, pragmatic trials are intended to answer the questions that clinicians and patients have regarding therapy.

The need for more representative subjects

Most RCTs use narrow inclusion and exclusion criteria to select the participants most likely to benefit from an intervention and least likely to experience harm (e.g. targeting those at high cardiovascular risk but excluding those with renal insufficiency). This approach maximizes the likelihood of observing benefit. However, this practice also results in many trials excluding subjects similar to the patients most commonly encountered in clinical practice. In a systematic sampling of RCTs published in high impact journals, 38.5% of RCTs excluded older adults, 81.3% excluded individuals with common medical conditions, and 54.1% excluded individuals receiving commonly prescribed medications.2

Considering the multiple morbidities present in 71% of diabetics, 82% of osteoarthritics, 83% of chronic obstructive pulmonary disease sufferers, and 92% of those with coronary artery disease,3 such trials are clearly not representative of real world populations.

Of 20,388 US Medicare patients ≥ 65 years of age, only 1 in 5 patients discharged from an acute care hospital with a diagnosis of congestive heart failure (CHF) met the criteria for enrollment in 3 landmark trials that guide the treatment of all CHF patients.4 As a general rule, although older adults and patients with multiple co-morbidities are often the target of clinical practice guidelines, they are poorly represented in the evidence-generating trials upon which clinical guidelines are based. This is especially important for the frail elderly, who fall into both categories. There are observational data raising questions as to the value of lowering blood pressure and blood sugar in the frail elderly.5, 6 Such studies are only hypothesis generating but clinical trials in this traditionally understudied population are clearly needed.

The need for better assessment of harm

The use of highly selected and generally healthier patients in efficacy trials increases the likelihood such trials will fail to adequately predict harm in a population with a broader spectrum of disease. This was well demonstrated following publication of the Randomized Aldactone Evaluation (RALES) trial when, coincident with increasing use of spironolactone in the general population, rates of hospitalization for hyperkalemia in Ontario rose from 2.4 per 1000 patients in 1994 to 11.0 per 1000 patients in 2001.7

The potential for adverse drug-drug or drug-disease interactions increases exponentially with each additional co-morbidity and each additional medication.8 It is reasonable to question whether the risk-benefit ratio of a new medication might differ meaningfully when that drug becomes the 7th or 8th in use by a patient with 4 co-morbidities as compared to a single drug in an uncomplicated patient.

Efficacy trials at times also employ run-in periods to exclude subjects who do not tolerate the drug, thus reducing the chances of detecting poor tolerability or harm. Given that clinical trials place less emphasis on the evaluation of adverse effects than on primary outcomes9, and given that adverse drug reactions are so varied in presentation (often mimicking medical diseases10), harmful effects of drugs may not be detected until years after a drug is approved for use.

The need for patient-oriented outcomes that matter

Much of the evidence in support of common interventions is based on surrogate outcomes such as HbA1c. This is problematic since recent studies challenge the assumption that surrogates can reliably predict the effect an intervention will have on "hard" outcomes.11 For example rosiglitazone lowers HbA1c but increases cardiovascular events.12 Complex clinical scales come somewhat closer to measuring the patient’s experience but these, often arbitrary combinations of clinical signs and symptoms, are also problematic in that their clinical meaning is often difficult to determine. As much as possible, outcome measures should be highly objective (e.g. mortality, disabling stroke, hospitalization, nursing home admission, hip fracture) and resonate with patients and clinicians alike (e.g. falls, cognitive impairment, independence, quality of life).

  1. Roland M, Torgerson DJ. What are pragmatic trials? BMJ. Jan 24 1998;316(7127):285.
  2. Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. Mar 21 2007;297(11):1233-1240.
  3. Information Services Division NHS Scotland. Measuring Long Term Health Conditions in Scotland. 2008; Accessed July 17 2013.
  4. Masoudi FA, Havranek EP, Wolfe P, et al. Most hospitalized older persons do not meet the enrollment criteria for clinical trials in heart failure. Am Heart J. Aug 2003;146(2):250-257.
  5. Odden MC, Peralta CA, Haan MN, Covinsky KE. Rethinking the association of high blood pressure with mortality in elderly adults: the impact of frailty. Arch Intern Med. Aug 13;172(15):1162-1168.
  6. Yau CK, Eng C, Cenzer IS, Boscardin WJ, Rice-Trumble K, Lee SJ. Glycosylated hemoglobin and functional decline in community dwelling nursing home-eligible elderly adults with diabetes mellitus. J Am Geriatr Soc. Jul;60(7):1215-1221.
  7. Juurlink DN, Mamdani MM, Lee DS, et al. Rates of hyperkalemia after publication of the Randomized Aldactone Evaluation Study. N Engl J Med. Aug 5 2004;351(6):543-551.
  8. Cadieux RJ. Drug interactions in the elderly. How multiple drug use increases risk exponentially. Postgrad Med. Dec 1989;86(8):179-186.
  9. Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. Jan 24-31 2001;285(4):437-443.
  10. Hilmer SN, McLachlan AJ, Le Couteur DG. Clinical pharmacology in the geriatric patient. Fundam Clin Pharmacol. Jun 2007;21(3):217-230.
  11. Yudkin JS, Lipska KJ, Montori VM. The idolatry of the surrogate. BMJ. Dec 28 2011;343:d7995.
  12. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med. Jun 14 2007;356(24):2457-2471.
* Originally published as Therapeutics Initiative Newsletter #88