B0557
Title: HODOR: Hold-Out Design for Online A/B testing with lurking variables
Authors: Nicholas Larsen - North Carolina State University (United States)
Jonathan Stallrich - North Carolina State University (United States)
Srijan Sengupta - North Carolina State University (United States)
Srijan Sengupta - North Carolina State University (United States) [presenting]
Abstract: A/B tests are common tools for estimating the average treatment effect in online controlled experiments (OCEs). Classical OCE theory is based on the Stable Unit Treatment Value Assumption, which holds that individual user responses are determined solely by the assigned treatment and not by the treatments of others. This assumption is violated when users are subjected to network interference, which is a common occurrence on social media platforms and other online testing platforms. Standard methods for estimating the average treatment effect typically fail to account for network effects, resulting in highly biased results. Furthermore, unobserved user covariates that influence user response and network structure, such as offline information or variables hidden due to privacy restrictions, can bias current estimators of the average treatment effect. The aim is to show how network-influential lurking variables can heavily bias popular network clustering-based methods, rendering them unreliable. To address this issue, we propose HODOR (Hold-Out Design for Online Randomized experiments), a two-stage design and estimation technique. We demonstrate that HODOR is unbiased for the average treatment effect and has low variance. Remarkably, HODOR provides reliable estimation even when the underlying network is partially unknown or uncertain.
