Nice paper this week in JAMA Internal Medicine. How to treat patients hospitalized with Community-Acquired Pneumonia (CAP)? Antibiotics, sure, but can you do more to improve outcome and shorten length of stay? You could choose any of 4 evidence-based interventions, that, according to (Cochrane) meta-analyses, improve patient outcome. Or decide to include all 4 in a bundle, as the Australian investigators did. And then the bundle fails to provide benefit and increases harm. Valentijn Schweitzer and I tried to explain.

The four measures, up and above antibiotics, were:

1. A fixed dosage of corticosteroids (50 mg for 7 days)
2. Early progressive mobilization of patients
3. Switching from parenteral to oral antibiotics upon predefined rules
4. Screening for malnutrition with targeted nutrition

As each of these interventions had level 1-2 evidence for improved outcome in patents with CAP implementing them was considered a “best-care” package, and with using a process-of-care approach to all patients a waiver of consent was granted, thereby maximizing representativeness and generalizability.

The Melbourne team wins the first price for catchy study design naming: an investigator-initiated, double-blind, stepped-wedge, cluster-randomized, Hybrid 1, effectiveness-implementation quality improvement trial! In other words: in 2 hospitals, each with 4 units, the bundle was implemented at regular intervals per unit and patient outcome was compared for the periods before and after implementation.

The goal was “to detect a clinically important decrease in the proportion of patients with a length of stay (LOS) greater than the median LOS from 36% to 20%”, which correspond to an Odds Ratio (OR) of 0.55

For simplicity I will focus only on the corticosteroid part of the bundle, as this is the most widely studied and most controversial measure and it is unlikely that the other 3 will reduce any benefits from the steroids.

So, the standard of care in these hospitals was that almost all patients with CAP were treated with 2 antibiotics (beta-lactam + macrolide/doxycycline), had a median LOS (before intervention) of 3 days (31% > 3 days) and day-30 mortality of 10.4%.

There were 415 and 401 patients in the before and after cluster. The bundle increased the proportion of patients receiving corticosteroids within 36 hours of admission from 25% to 73%, and the proportion that received steroids for at least 7 days increased from 3% to 56%. All other interventions also increased. So, this is what the real-world looks like, if you implement a bundle, rather than select (or cherry-pick) patients for a RCT.

The effects on outcome were disappointing (I guess). The OR was 0.95 (95% CI 0.57-1.59) for LOS and 0.78 (95% CI 0.38-1.61) for day-30 mortality. Actually, the proportion of patients with a LOS greater than the median LOS (3 days) increased from 31% to 35% (thus did not decrease). The bundle was associated with more hyperglycemia and gastrointestinal bleeding.

So, is this a negative or a non-conclusive study?

Despite the catchy (and excellent) design and the bravery to do a real-world study it is not large enough to be conclusive (which means that it is non-conclusive).

The confidence interval of the OR for day-30 mortality still holds a 50% reduction, which few would consider not to be relevant. To illustrate which effects are still plausible in this study, Bayesian analyses might help, as these estimate the likelihood of a certain effect (i.e. OR of 0.5 or more) given the data in the study. That would provide a better feeling whether clinically relevant effects are still plausible.

As it is, you could say that the results serve all: If you are using steroids in CAP you may say “benefits not excluded”, and if you’re not using steroids these findings will not enforce you to change practice.

The best aspect of this study, though, is that “efficacy of individual interventions demonstrated in clinical trials may not necessarily translate into effectiveness when implemented in combination and may even result in net harm.” For instance for corticosteroids, the authors state that we have “at least 17 RCTs (with >2000 participants) … subjected to numerous meta-analyses by separate research groups, yielding consistent results interpreted in a similar fashion. The recent Cochrane review states that “people with CAP treated with corticosteroids had lower clinical failure rates (death, worsening of imaging studies, or no clinical improvement), shorter time to cure, shorter hospital stay, and fewer complications.” Meta-analyses have suggested that overall rates of serious adverse events are no higher or are even reduced with corticosteroids.”

And only RCTs and meta-analyses can yield a level 1-2 evidence recommendation for our practice. This study suggests that – at least for these interventions  – RCT results may not lead to similar effects when implemented for all patients. Yet, with a larger sample, the effect might have been different. Also, what would have happened if they had managed to increase the proportion with 7 days on steroids, from 3% to 80% instead of the 3% to 56% observed in the trial? Would this further increase the trend of a reduction in mortality and LOS? We will not know.

What we did not understand is how on earth did they manage to make this a double-blind study? Apparently they didn’t tell patients about the study and didn’t disclose the allocation code to the statistician. Let’s call that a real-world double-blinding.