Ready to Roll? Practical Guidance on Whether and When to Aggregate Data in Health Policy Evaluation

Abstract

Health policy researchers often have questions about the effects of state policy on individual-level outcomes collected over multiple time periods. For example, limited evidence suggests that medical cannabis may be an effective substitute for opioids in pain management, which raises a question about the effect of medical cannabis laws on receipt of opioid treatment among individuals with chronic non-cancer pain. This question might be addressed using, for example, a large health insurance claims database which would track individual's receipt of such treatment. An open question in this setting is whether the researcher can or should "roll-up" (i.e., aggregate, average, or pool) this individual-level data to the state level when assessing the effects of state policy. Rolling up the data offers a clear computational advantage since it makes the individual-level big data question much smaller. However, existing literature does not sufficiently address whether and when aggregation is disadvantageous due to loss of individual-level information.

Here, we examine the statistical performance of a variety of common methods in health policy evaluation (two-way fixed effects, difference-in-differences with staggered adoption methods, trial emulation, and marginal structural models) which permit the use of either individual- or aggregate-level data to offer practical guidance on whether and when to roll up. Our guidance is based on simulation models which allow us to make fair comparisons between analytic methods under a variety of controlled conditions. We also discuss our recommendations in the context of a study designed to assess the effects of state medical cannabis laws on opioid prescribing among patients with chronic non-cancer pain. Preliminary results indicate that rolling up does not yield meaningful loss of statistical efficiency in simple settings in which the analysis does not take advantage of individual-level data. When reasonable, aggregation allows analysts to spend less time with specialized big data tools and focus more on efficiently moving through the data-to-policy pipeline.