Ready to roll? Practical guidance on whether and when to aggregate data in health policy evaluation

Abstract

Research Objective: Health policy researchers often have questions about the effects of state policy on individual-level outcomes collected over multiple time periods. For example, limited evidence suggests that medical cannabis may be an effective substitute for opioids in pain management, which raises a question about the effect of medical cannabis laws on receipt of opioid treatment among individuals with chronic non-cancer pain. This question might be addressed using, for example, a large health insurance claims database which would track individuals’ receipt of such treatment. An open question in this setting is whether the researcher can or should “roll-up” (i.e., aggregate, average, or pool) this individual-level data to the state level when assessing the effects of state policy. Rolling up the data offers a clear computational advantage since it makes the individual-level big data question much smaller. However, existing literature does not sufficiently address whether and when aggregation is disadvantageous due to loss of individual-level information.

Study Design: We examine the statistical performance of a variety of common methods in health policy evaluation (two-way fixed effects, difference-in-differences with staggered adoption methods, trial emulation, and marginal structural models) which permit the use of either individual- or aggregate-level data to offer practical guidance on whether and when to roll up. Our guidance is based on simulation models which allow us to make fair comparisons between analytic methods under a variety of controlled conditions.

Population Studied: We discuss our recommendations in the context of a study designed to assess the effects of state medical cannabis laws on opioid prescribing among patients with chronic non-cancer pain. The study sample consists of individuals 18+ with a chronic non-cancer pain condition who live in a “treated” (i.e., law-enacting) or “comparison” (no law) state and are continuously enrolled in a UnitedHealthcare insurance plan for the 7 years around the treated state’s implementation of the law.

Principal Findings: Preliminary results indicate that rolling up does not yield loss of statistical efficiency in simple settings in which the analysis does not sufficiently take advantage of individual-level data and when standard errors are clustered at the state level.

Conclusions: Investigators and analysts may, in some settings, be able to roll up individual level data for computational efficiency without meaningful loss of statistical efficiency.

Implications for Policy or Practice: This work seeks to reduce uncertainty among investigators and analysts about whether and how to leverage individual-level information in policy evaluation data.