Skip to main content

Ranking the effectiveness of worldwide COVID-19 government interventions

Our take —

This study used four different analytic approaches to estimate the relative effectiveness of country- and US state-level non-pharmaceutical interventions (NPIs) early in the pandemic. The authors found that social distancing (e.g., cancelling small gatherings), school closures, and travel restrictions (e.g., border closures) were most effective, while environmental measures (e.g., disinfecting surfaces) were least effective. Though each statistical approach was distinct, all relied on a similar set of assumptions, and all were subject to biases. Assessing the effectiveness of NPIs is difficult for many reasons: many were implemented simultaneously, case reporting varies across regions and over time, and the success of a given NPI may heavily depend upon local context. This study is not definitive, but suggestive; it highlights important issues involved with NPI impact evaluation.

Study design

Ecological; Modeling/Simulation

Study population and setting

This study considered 6,068 non-pharmaceutical interventions (NPIs) in 79 territories (countries and US states) implemented in March and April of 2020. The authors ranked the effectiveness of these NPIs in reducing the effective reproduction number of SARS-CoV-2 (Rt) using eight broad themes, and 46 categories of NPIs that were implemented 5 or more times within those themes, established by the Complexity Science Hub COVID-19 Control Strategies List (CCCSL). Four analytic methods were used to rank effectiveness: 1) a case-control study (similar to a difference-in-differences approach), 2) LASSO time-series regression (an approach designed to prevent overfitting of models), 3) random forest regression (a a prediction and classification method) with NPIs ranked by measuring model performance with and without each NPI, and 4) Transformers, a machine learning technique for parallel processing of sequential data that can be applied to time series. All approaches used a range of lag periods (i.e., the timing between implementation of an intervention and its effect on the reproductive number, Rt), and considered possible confounding variables including the duration of the local epidemic, Rt at the time of a given NPI, population size, population density, and the previous number of NPIs implemented (total and within the same category as the given NPI). The results were applied to two external validation data sets.

Summary of Main Findings

There was general agreement among the four methods used to rank NPI effectiveness; of the eight broad NPI themes, social distancing and travel restrictions were the highest ranked, while environmental measures such as disinfecting surfaces were least effective. Six of the 46 categories showed statistically significant reductions in the reproductive number (Rt) with all four methods: canceling small gatherings (change in Rt: -0.22 to -0.35), closing educational institutions (-0.15 to -0.21), border restrictions (-0.06 to -0.23), increasing availability of personal protective equipment (-0.06 to -0.13), individual movement restrictions (-0.08 to -0.13), and national lockdown including U.S. state stay-at-home orders (-0.01 to -0.14). Environmental cleaning, public transport restrictions, and contact tracing measures were among the least effective NPIs across estimating methods. NPI effectiveness was highly heterogeneous across countries. Applying these methods to two external datasets produced broadly similar results. By artificially shifting NPIs to different times in relation to the age of the epidemic, the authors estimated that early adoption was beneficial for lockdown, small gathering cancellations, travel restrictions, and closure of educational institutions.

Study Strengths

This study considered a wide range of disaggregated NPIs, and compared their effectiveness using four distinct methods to examine the sensitivity of NPI ranking to the estimation method. Methods were tested on two external datasets.

Limitations

While the authors employed four different methods for estimating NPI effectiveness, the specification of models in each method was structurally similar, meaning that agreement across methods is less remarkable than it may appear. The estimation of Rt relies on reported cases; case reporting was highly heterogeneous across regions and over time during the study period, and was subject to discrete changes in case definitions and reporting standards. Estimates of changes in Rt may therefore be biased. Also, using this method makes it appear as if measures that increase case ascertainment (e.g., improving testing capacity, contact tracing, etc.) actually increase Rt. The timing of NPI implementation makes identification of impacts difficult, since so many were implemented roughly simultaneously– this issue is exacerbated because the expected lags between implementation and effects are not precisely known. The study is restricted to March and April of 2020, and so territories that were hit earlier by the pandemic are likely overrepresented. Although travel restrictions are ranked highly, they are likely to be effective only during the initial phases of an epidemic when local transmission represents a small share of new cases. In general, the interactions between the age of a local epidemic and NPI effectiveness are not handled well by the approaches here. Finally, although the authors adjusted for several territory-level variables, including the number of previous interventions implemented, residual confounding (e.g., country GDP) and unmeasured interactions (e.g., the success of contact tracing may depend on multiple other factors including public health infrastructure and testing capacity) are likely.

Value added

This is one of the largest attempts to quantify and rank the effectiveness of non-pharmaceutical interventions across multiple regions.  

This review was posted on: 14 December 2020