To kick off the 2018 Journal Club our PhD students discussed a bewildering new trial design* to determine vaccine effectiveness (VE) published in Lancet ID, from which Meri Varkila reports.  The classical approach to quantify VE was to spend the best 5 years of your life to find 2,000 general practitioners, to invite 600,000 elderly to randomize 85.000 and to find 139 primary endpoints in 57 hospitals while all involved remain blinded. This new approach, called the test-negative design (TND) study would give you that number in a year, by just studying a few hundred patients with community-acquired pneumonia. A true Quality-of-Life enhancer for many…., if reliable.

In short the unanswered scientific question: Is the pneumococcal conjugate vaccine, PCV13, more effective in preventing pneumococcal infections than the older 23-valent polysaccharide vaccine (PPV23). If so, should the more expensive PCV13 be recommended to all elderly?

To a group well-acquainted with the challenges of VE research, this new trial design sounded almost too good to be true: You classify your patients that enter your Emergency Room with pneumococcal pneumonia as cases (test-positives) and those with pneumonia from other causes as controls (test-negatives), and collect their pneumococcal vaccination status (yes or no, just by asking) and known risk factors for pneumonia. Then determine VE by calculating the ratio of being vaccinated among cases versus the odds of being vaccinated among controls from the observational data.  In addition to being cost- and time-efficient, “TND reduces misclassification and confounding/selection bias due to health-care-seeking behavior”, according to the authors, which usually makes the observational design non-suited. “Wow”, thought the students, “If all of this is true, why would anybody ever conduct an RCT, cohort or case-control study again?”.

But as they say, if something sounds too good to be true, it probably is. A recently posted paper on a prepint server attemptesd to determine the accuracy of the NTD from a more mathematical angle and found that TND should meet two assumptions in order to measure true vaccine effects based on odds ratios: (1) vaccination should be uncorrelated with exposure and susceptibility to infection and severity of symptoms and, (2) vaccination should confer “all-or-nothing” protection. These assumptions raised some concerns within the journal club. All study subjects had clinical criteria for suspected pneumonia, but differences in risk factors, severity of disease, and propensity for vaccination between vaccinated and unvaccinated subjects were not excluded, as baseline characteristics were not provided. The second assumption is unknown and seems definitely not to hold for pneumococcal polysaccharide vaccines concluded the group. With some possibility of misclassification added, the students’ enthusiasm about TND started to fade. The authors of the preprint paper stated “Our findings demonstrate a need to reassess how data from test-negative studies are interpreted for policy decisions conventionally based on causal inferences.”

In conclusion, although not entirely convinced by the evidence provided by this TND study, the journal club group commends Suzuki et al. for their creative approach to investigating this difficult question. The reported VE of PPV23 against all pneumococcal pneumonia of 27.4% (95%CI 3.2-45.6) and of 33.5% (95%CI 5.6-53.1) against PPV23 serotypes , would make this vaccine equally effective as PCV13, where VE came from the “little more laborious” RCT. As the raging debate regarding the best pneumococcal vaccine for elderly continues in our Health Council, the students decided to put their money on PCV13. And as for the best study design for vaccine effectiveness? Definitely the RCT.

* What we considered “new” appeared to be relatively new, with the first study being from 1980! (thanks to Marc Lipsitch)

Meri Varkila is doing PhD research on sepsis in the Intensive Care Unit within the framework of the MARS (Molecular Diagnosis and Risk Stratification of Sepsis) project.