Study population and setting
This study considered 6,068 non-pharmaceutical interventions (NPIs) in 79 territories (countries and US states) implemented in March and April of 2020. The authors ranked the effectiveness of these NPIs in reducing the effective reproduction number of SARS-CoV-2 (Rt) using eight broad themes, and 46 categories of NPIs that were implemented 5 or more times within those themes, established by the Complexity Science Hub COVID-19 Control Strategies List (CCCSL). Four analytic methods were used to rank effectiveness: 1) a case-control study (similar to a difference-in-differences approach), 2) LASSO time-series regression (an approach designed to prevent overfitting of models), 3) random forest regression (a a prediction and classification method) with NPIs ranked by measuring model performance with and without each NPI, and 4) Transformers, a machine learning technique for parallel processing of sequential data that can be applied to time series. All approaches used a range of lag periods (i.e., the timing between implementation of an intervention and its effect on the reproductive number, Rt), and considered possible confounding variables including the duration of the local epidemic, Rt at the time of a given NPI, population size, population density, and the previous number of NPIs implemented (total and within the same category as the given NPI). The results were applied to two external validation data sets.
Summary of Main Findings
There was general agreement among the four methods used to rank NPI effectiveness; of the eight broad NPI themes, social distancing and travel restrictions were the highest ranked, while environmental measures such as disinfecting surfaces were least effective. Six of the 46 categories showed statistically significant reductions in the reproductive number (Rt) with all four methods: canceling small gatherings (change in Rt: -0.22 to -0.35), closing educational institutions (-0.15 to -0.21), border restrictions (-0.06 to -0.23), increasing availability of personal protective equipment (-0.06 to -0.13), individual movement restrictions (-0.08 to -0.13), and national lockdown including U.S. state stay-at-home orders (-0.01 to -0.14). Environmental cleaning, public transport restrictions, and contact tracing measures were among the least effective NPIs across estimating methods. NPI effectiveness was highly heterogeneous across countries. Applying these methods to two external datasets produced broadly similar results. By artificially shifting NPIs to different times in relation to the age of the epidemic, the authors estimated that early adoption was beneficial for lockdown, small gathering cancellations, travel restrictions, and closure of educational institutions.
This study considered a wide range of disaggregated NPIs, and compared their effectiveness using four distinct methods to examine the sensitivity of NPI ranking to the estimation method. Methods were tested on two external datasets.
While the authors employed four different methods for estimating NPI effectiveness, the specification of models in each method was structurally similar, meaning that agreement across methods is less remarkable than it may appear. The estimation of Rt relies on reported cases; case reporting was highly heterogeneous across regions and over time during the study period, and was subject to discrete changes in case definitions and reporting standards. Estimates of changes in Rt may therefore be biased. Also, using this method makes it appear as if measures that increase case ascertainment (e.g., improving testing capacity, contact tracing, etc.) actually increase Rt. The timing of NPI implementation makes identification of impacts difficult, since so many were implemented roughly simultaneously– this issue is exacerbated because the expected lags between implementation and effects are not precisely known. The study is restricted to March and April of 2020, and so territories that were hit earlier by the pandemic are likely overrepresented. Although travel restrictions are ranked highly, they are likely to be effective only during the initial phases of an epidemic when local transmission represents a small share of new cases. In general, the interactions between the age of a local epidemic and NPI effectiveness are not handled well by the approaches here. Finally, although the authors adjusted for several territory-level variables, including the number of previous interventions implemented, residual confounding (e.g., country GDP) and unmeasured interactions (e.g., the success of contact tracing may depend on multiple other factors including public health infrastructure and testing capacity) are likely.
This is one of the largest attempts to quantify and rank the effectiveness of non-pharmaceutical interventions across multiple regions.
This review was posted on: 14 December 2020