Deeks-etal-2024-lessons-to-be-learned-from-test-evaluations-during-the-COVID-19-pandemic
The Royal Statistical Society (RSS) Working Party, led by Jonathan J. Deeks, produced a comprehensive report analysing statistical principles, practical challenges, and regulatory issues surrounding the evaluation of in‑vitro diagnostic (IVD) tests during the COVID‑19 pandemic. The document begins with an infectious‑disease background and defines key diagnostic concepts such as analytical versus clinical performance, sensitivity, specificity, predictive values, limits of detection/quantification, and the relationship between RT‑PCR cycle‑threshold (Ct) values, viral load, and infectiousness. It surveys a range of study designs—analytical, two‑gate (case‑control), prospective field, paired, and serial‑testing approaches—highlighting bias sources (patient selection, reference‑standard imperfections, operator effects, flow‑and‑timing) and sample‑size considerations, especially in low‑prevalence settings. The report details empirical lessons from COVID‑19 testing: the necessity of explicit intended‑use definitions, performance variability across populations, specimen types, and operators; the pitfalls of relying on analytical data or spiked samples for regulatory approval; and the difficulty of defining infectiousness without a gold‑standard, illustrated by Ct‑value and viral‑culture studies. Communication challenges are examined, showing how mis‑interpretation of sensitivity, specificity, and predictive values can mislead the public and policymakers. Advanced design issues such as head‑to‑head test comparisons, efficient sampling (testing all positives and a random subset of negatives), and the methodological complexities of test‑intervention randomized trials are discussed. Regulatory observations cover the EU‑IVD Regulation, UK MHRA processes, Emergency Use Authorisations, Target Product Profiles, and the need for post‑marketing surveillance. The authors stress transparency: public registration of protocols, adherence to STARD/PRISMA‑DTA reporting, open data sharing, and systematic reviews to inform policy. The report culminates in 22 concrete recommendations grouped under Study‑Design, Regulation, and Transparency, aiming to improve future diagnostic test evaluation, regulatory oversight, and public confidence, with relevance beyond COVID‑19.
Topics
Motivation, Scope, and Infectious‑Disease Background of the RSS Working Party
Explains why the Royal Statistical Society convened a biostatistics working group in July 2020, outlining the COVID‑19 pandemic context, the need for robust diagnostic test evaluation, and the group’s objectives to improve statistical practice and public confidence.
Diagnostic Terminology and Statistical Estimands for IVD Evaluation
Defines key concepts such as analytical vs. clinical performance, sensitivity, specificity, predictive values, limits of detection/quantification, confidence intervals, bias, and precision, establishing the statistical language used throughout the report.
Study‑Design Options and Advanced Design Considerations
Compares analytical studies, two‑gate (case‑control), prospective field, paired, serial‑testing, head‑to‑head comparisons, and efficient sampling schemes (testing all positives plus a random subset of negatives). Discusses alignment with intended‑use, randomisation of swab order, and design choices for unbiased performance estimation.
Bias Sources, Flow‑and‑Timing Issues, and Sample‑Size Planning
Details patient‑selection bias, reference‑standard imperfections, operator effects, timing of specimen collection, flow‑and‑timing problems, and provides formulas and guidance for determining adequate sample sizes, especially in low‑prevalence settings.
Ct Values, Viral Load, and Infectiousness: Measurement Challenges
Explores the relationship between RT‑PCR cycle‑threshold values, viral load, and culture positivity, highlighting lack of standardisation, statistical modelling difficulties, and implications for interpreting antigen‑test performance and defining infectiousness.
Communicating Predictive Values and Test Errors to the Public
Provides guidance on how to explain sensitivity, specificity, PPV, NPV, and likelihood ratios to lay audiences, illustrating common mis‑interpretations in policy guidance (e.g., school testing) and the impact of prevalence on perceived test accuracy.
Test‑Strategy Randomised Trials and Methodological Challenges
Reviews RCTs and observational studies of mass‑testing, school testing, and contact‑release strategies, addressing issues of bias, clustering, large sample requirements, outcome measurement, and the integration of diagnostic performance into intervention effectiveness.
Regulatory Frameworks for IVDs During the Pandemic
Analyzes the EU‑IVD Regulation, UK MHRA processes, CE marking, Emergency Use Authorisations, Target Product Profiles, and the role of statistical expertise in licensing decisions, highlighting gaps revealed by COVID‑19 test approvals.