Study population and setting
The state of California issued a shelter-in-place order (SIPO) on March 19, 2020, two days after several counties and municipalities in CA had begun to issue similar orders. The order required individuals to stay at home or in their place of residence, with exceptions made for critical infrastructure sectors (e.g., health care, food, agriculture, public safety, and media) and activities deemed essential (e.g., grocery shopping, pharmacy visits, and exercise). The authors estimated the effects of the SIPO on mobility, COVID-19 case counts, and mortality using a synthetic control modeling approach; this approach generates a counterfactual proxy for what would have happened in CA without the SIPO by weighting outcomes experienced by other states that did not implement a SIPO, or that did so after California.
Summary of Main Findings
From the authors’ primary estimates, California’s SIPO was associated with a 2.1% increase in the proportion of individuals staying at home per day, a total of 49,926 averted COVID-19 cases, and a total of 763 fewer COVID-19-related deaths relative to its synthetic control (a modeled version of California without the SIPO) between March 19, 2020 and April 20, 2020. Results for mobility and case counts were statistically significant (p<0.05 from permutation-based tests), while the result for mortality was not. Altering the criteria for states serving as donors for the control, and changing other assumptions of the model, did not substantially change results. The authors conclude that the SIPO reduced mobility and COVID-19 cases and deaths.
The timing of California’s SIPO afforded a long enough follow-up period to accumulate a large number of post-intervention outcomes. The analytic approach used (synthetic controls) makes it explicit which states are being used as comparison units and how the counterfactual is constructed. The authors performed some robustness checks with alternate methods and assumptions and found results were generally consistent.
To construct the counterfactual proxy (i.e., what California’s experience would have been in the absence of the SIPO), the authors included data from 32 states that implemented SIPOs at least five calendar days after California did. The counterfactual being modeled is thus an ill-defined mixture of no SIPO and delayed SIPO, which the authors suggest may represent a lower bound for the effects of the SIPO relative to none. Stricter criteria for including other states’ data resulted in stronger effect estimates but more uncertainty. It is unclear why a relatively small increase (2.1%) in the shelter-in-place index would result in such large impacts on case counts and mortality, which may indicate issues with either the index itself or the validity of the case count and mortality results. Specific concerns with the synthetic control approach as implemented include 1) matching on a limited number of pre-period observations, 2) no placebo checks to assess the stability of the estimator over time, and 3) highly variable donor weights/estimator results across changes to the specification, suggesting that the results may be quite sensitive.
This study is the first to provide evidence that the early shelter-in-place order implemented by California was effective in reducing transmission of SARS-CoV-2 and consequent mortality.