Skip to main content

Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: An observational cohort study

Our take —

Twenty-two COVID-19 prognostic models for clinical deterioration or mortality were systematically and externally validated with an independent dataset of 411 COVID-19 patients in London. Although the NEWS2 and final Carr model performed relatively well in this population, single-variable models with baseline oxygen saturation and age were the strongest predictors of clinical deterioration and morality, respectively. However, this was a single-site validation study, evaluating prognostic models developed prior to May 5, 2020, and multi-site studies and systematic evaluation of newer models are necessary.

Study design

Retrospective Cohort, Other

Study population and setting

This study externally validated 22 prognostic models aiming to predict clinical deterioration or death among COVID-19 patients with an independent dataset of 411 patients admitted to the University College Hospital London from February 1 to April 30, 2020. The 22 prognostic models were selected from a rigorously conducted systematic review of COVID-19 prognostic models published through May 5, 2020. Participants were included in the external validation if they had PCR-confirmed SARS-CoV-2 or clinically diagnosed COVID-19 within 5 days of hospital admission; patients transferred from other hospitals were excluded.

Summary of Main Findings

Among the 411 participants in the external validation dataset (median age 66 years, 61% male, 90% PCR-confirmed), 180 had an outcome of clinical deterioration (death or initiation of ventilation support other than standard oxygen therapy), of whom 115 died. The best performing models were the NEWS2 score for predicting deterioration within 24 hours and the final model by Carr and colleagues that predicted deterioration over 14 days, both of which achieved an area under the receiver operating characteristic curve (AUROC) of 0.78 and appeared relatively well calibrated (agreement between observed and predicted outcome risk). Admission oxygen saturation and age were the strongest univariate predictors of clinical deterioration and mortality, respectively, with AUCs of 0.76. In decision curve analyses, none of the prognostic models had greater net benefit than oxygen saturation for predicting clinical deterioration or age for predicting mortality.

Study Strengths

Missing data were multiply imputed. Results were consistent across separate sensitivity analyses that excluded patients without PCR-confirmed SARS-CoV-2 infection, excluded patients clinically deteriorated within 4 hours of hospital admission, and excluded patients with any missing data.

Limitations

This was a single-site validation, and it is not clear how comparable this study population is to either the study populations from the original studies or their intended target populations; multi-site validation studies are needed. The study only included prognostic models developed during the early phase of the pandemic (prior to May 5, 2020), and almost half were developed with datasets from China (n=10), so the degree to which these results apply to current COVID-19 outcomes or study populations is unclear. There was substantial missing data for lactate dehydrogenase and D-dimer, and though it was multiply imputed, future studies with more complete data will help to identify the importance of these predictors. Several models could not be replicated with the external validation dataset due unavailability of certain data (e.g., imaging).

Value added

This is the first systematic external validation of multiple existing prognostic models for COVID-19 outcomes using an independent dataset.

This review was posted on: 10 November 2020