Skip to main content

Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19

Our take —

The authors present COVID-GRAM, a web-based application that aims to predict incident critical illness among patients hospitalized for COVID-19. Considering several limitations, including bias due to overfitting, and the need for validation in settings outside of China, the risk score is a promising point-of-care tool for identifying patients at hospital admission who are most likely (or unlikely) to develop critical illness, which can aid in decision-making about how to best allocate resources.

Study design

Retrospective cohort

Study population and setting

This study developed and validated a risk score for critical COVID-19 illness, defined as the composite of admission to the ICU, invasive ventilation, or death. The development cohort included 1590 patients (mean age 49 years, 57% male, 25% with coexisting condition) with lab-confirmed SARS-CoV-2 infection admitted to 575 hospitals in China between November 21, 2019 and January 31, 2020. The validation cohort (n=710; mean age 48 years, 54% male, 24% with coexisting condition) included data from four sources, three of which had follow-up through February 28, 2020. Seventy-two variables, including clinical signs and symptoms, imaging results, laboratory findings, demographic variables, and medical history, measured at hospital admission were considered.

Summary of Main Findings

Eight percent (n=131) of patients in the development cohort and 12% (n=87) of patients in the validation cohort developed critical illness. Using LASSO (least absolute selection and shrinkage operator) regression, the authors narrowed 72 candidate variables down to 19. The 19 variables were then put into a logistic regression model, and the 10 that remained significant (p<0.05) were included in the risk score (chest radiography abnormality, age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil-lymphocyte ratio, lactate dehydrogenase, and direct bilirubin), which is available as a web-based tool. The mean AUC (a measure of how well the model can distinguish between persons with and without the outcome) in both the development and validation cohorts was 0.88.

Study Strengths

The study developed an easy-to-use web-based calculator using 10 variables that are typically available at the time of hospital admission. Multiple imputation was used for variables with missingness <20% The data were independently reviewed and verified by two clinicians. Validation was done in an independent and external dataset.


The data for development and validation cohorts were from China, so the applicability of the model to populations outside of China is unknown. It is unclear whether the authors corrected for overfitting; they use the bootstrap to estimate mean AUC over 200 bootstraps, but this metric should be corrected for overfitting, and failure to do so can lead to overly optimistic predictions. Data were collected during the early stages of the pandemic in China, and hospital practices may differ from current guidelines. The authors assumed linear relationships for continuous predictors, which is unlikely.

Value added

This paper presents an easy-to-use web-based application to predict risk of critical illness among patients hospitalized for COVID-19.