Clustered Switchback Designs for Experimentation Under Spatio-temporal Interference

Su Jia; Nathan Kallus; Christina Lee Yu

math.STDec 25, 2023

Clustered Switchback Designs for Experimentation Under Spatio-temporal Interference

Authors: Su Jia, Nathan Kallus, Christina Lee Yu

Organizations: Cornell University

Abstract

We consider experimentation in the presence of non-stationarity, inter-unit (spatial) interference, and carry-over effects (temporal interference), where we wish to estimate the global average treatment effect (GATE), the difference between average outcomes having exposed all units at all times to treatment or to control. We suppose spatial interference is described by a graph, where a unit's outcome depends on its neighborhood's treatments, and that temporal interference is described by an MDP, where the transition kernel under either treatment (action) satisfies a rapid mixing condition. We propose a clustered switchback design, where units are grouped into clusters and time steps are grouped into blocks, and each whole cluster-block combination is assigned a single random treatment. Under this design, we show that for graphs that admit good clustering, a truncated Horvitz-Thompson estimator achieves a $\tilde O(1/NT)$ mean squared error (MSE), matching the lower bound up to logarithmic terms for sparse graphs. Our results simultaneously generalize the results from \citet{hu2022switchback,ugander2013graph} and \citet{leung2022rate}. Simulation studies validate the favorable performance of our approach.

Figures & tables

Figure 1: Clustered Switchback Experiments . The image illustrates clustered switchback on DoorDash [ 34 ] . The time and geographical locations are grouped into blocks. Each spatio-temporal cluster (i.e., Cartesian product) is independently assigned a treatment arm. The goal is to estimate the difference in the average (counterfactual) “outcomes” (e.g., revenues) between the all-treatment and all-control policy.

Interference Graph	(Spatio-) Clustering	Timeblocks	MSE
No edges (no interference)	one node per cluster	$r=\tilde{O}(t_{\rm mix})$	Corollary 2 : $\widetilde{O}\left(\frac{t_{\rm mix}}{TN}\right)$
Arbitrary	all nodes in single cluster (pure switchback)	$r=\tilde{O}(t_{\rm mix})$	Corollary 3 : $\widetilde{O}\left(\frac{t_{\rm mix}}{T}\right)$ (also Hu and Wager [16] )
Max Degree $D$	max cluster degree $d_{\Pi}^{\rm max}$ max cluster size $N_{c}$	$r=\tilde{O}(t_{\rm mix})$	Corollary 4 : $\widetilde{O}\left(\frac{t_{\rm mix}Dd_{\Pi}^{\rm max}N_{c}2^{2d_{\Pi}^{\rm max}}}{TN}\right)$
2D-grid spatial graph edges w/in distance $h$	uniform 2D grid	$r=\tilde{O}(t_{\rm mix})$	Corollary 6 : $\widetilde{O}\left(\frac{t_{\rm mix}h^{2}}{TN}\right)$
2D-grid spatial graph edges w/in distance	uniform 2D grid	$r=T$ (pure GCR)	Leung [23] : $\widetilde{O}\left(\frac{h^{2}}{N}\right)$
$\kappa$ -Restricted Growth Max Degree $D$	$1$ -hop-max randomized clustering	$r=\tilde{O}(t_{\rm mix})$	Corollary 7 : $\widetilde{O}\left(\frac{t_{\rm mix}D^{3}\kappa^{5}}{TN}\right)$

Table 1: Implications of Our Main Result alongside Known Results: Our main theorem implies the following MSE bounds in several fundamental special cases. We omit polylogarithmic terms in

N,T

in these MSE bounds. We also show our results alongside several known results for “pure” switchback and “pure” A/B testing under interference.

Figure 2: Positioning of this work. By and large, a model for experimentation involves a subset of four key features, as illustrated above. Prior work has addressed spatial and temporal interference separately, often incorporating heterogeneity across individuals as well. However, in real-world applications, all four features often arise simultaneously. This work aims to lay the theoretical foundation for cluster switchback experiments, which are increasingly used in practice to navigate such complex interference.

Figure 3: Dependence Graph: The regions correspond to the clusters in a partition $\Pi$ . Units $i,j$ intersect a common cluster $C$ , so $(i,j)\in E_{\Pi}$ (or $i\not\perp\!\!\!\perp j$ ).

Design	When to Use	MSE bound
Pure Switchback	High temporal mixing; environment stable over time	$\tilde{O}\left(\frac{t_{\rm mix}}{T}\right)$
Pure Spatial Clustering	Strong spatial interference; sparse or weakly connected graph	$O\left(\frac{\|\{(i,i^{\prime}):i\not\perp\!\!\!\perp i^{\prime}\}\|}{2^{-2d_{\Pi}^{\rm max}}N^{2}}\right)$
Clustered Switchback	Both significant spatial interference and fast temporal dynamics	$\tilde{O}\left(\frac{t_{\rm mix}}{T}\cdot\frac{\|\{(i,i^{\prime}):i\not\perp\!\!\!\perp i^{\prime}\}\|}{2^{-2d_{\Pi}^{\rm max}}N^{2}}\right)$

Table 2: When to use each experimental design.

Figure 4: Locations of 169 Hotels in Washington DC provided in dataset [ 8 ] .

Statistic	Value
Nodes	169
Edges	4,332
Mean degree	51.3
Median degree	56
Std. degree	30.3
Min / Max degree	0 / 95

Table 3: Summary statistics of the DC hotel interference graph (

\kappa=0.1

Figure 7: MSE vs. total (spatio-temporal) clusters $K$ , with HT estimator, $\delta=0.2$ . Joint spatio-temporal design has lower MSE than both pure designs.

Figure 8: Heatmaps of bias (left), standard deviation (centre), and MSE (right) for the HT estimator as a function of time-block length $\ell$ (rows) and fractional recency $f_{r}=r/\ell$ (columns), fixing $1/s=4$ and $\delta=0.2$ (Uniform network, $n=100$ , $T=10{,}000$ ). MSE is U-shaped in $f_{r}$ for every $\ell$ , with the optimum near $f_{r}^{*}\approx 0.5$ .

Figure 9: Heatmaps of bias (left), standard deviation (center), and MSE (right) for the HT estimator as a function of spatial cells per dimension $1/s$ (rows) and fractional recency $f_{r}$ (columns), fixing $\ell=100$ and $\delta=0.2$ (Uniform network). MSE is minimized at an interior value $1/s\in\{3,4\}$ ; beyond this, variance rises as each spatial cluster contains fewer units.

Figure 10: Heatmaps of bias, standard deviation, and MSE for the HT estimator over $(\ell,1/s)$ ( $\delta=0.2$ , best $r$ per cell). Rows: $1/s\in\{1,\ldots,8\}$ ; columns: $\ell\in\{40,\ldots,1000\}$ . This is expected, since when $\ell$ is small, “contaminated” data (rounds) make up a higher proportion of the entire data.

Figure 11: MSE vs. time-block length $\ell$ for the Hajék (left), HT (center), and DM (right) estimators, fixing $1/s=3$ and $\delta=0.2$ (Uniform network, $x$ -axis truncated to $\ell\leq 400$ , $y$ -axis to $[0,0.5]$ ). Each curve corresponds to a different fractional parameter: $f_{r}\in\{0.25,0.5,1,1.5,2\}$ for Hajék/HT and $f_{b}\in\{0,0.1,0.3,0.5\}$ for DM. Hajék uniformly dominates HT; DM with $f_{b}=0$ is competitive at large $\ell$ but biased at small $\ell$ .

Theoretical prediction	Empirical finding	Status
Joint $\succ$ pure temporal	Joint MSE $2.4\times$ lower	✓ Confirmed ( Fig. 7 )
Joint $\succ$ pure spatial	Joint MSE $28\times$ lower	✓ Confirmed ( Fig. 7 )
HT $\succ$ DM (asymptotically)	HT $\succ$ DM only at small $\ell$	$\triangle$ Finite-sample caveat ( Fig. 11 )
MSE U-shaped in $\ell$	Sweet spot at $\ell\approx 200$ – $400$	✓ Confirmed ( Figs. 10 and 11 )
Bias–variance trade-off in $r$	U-shaped MSE vs. $r$	✓ Confirmed ( Fig. 8 )
Hajék $\approx$ HT variant	Hajék: median $60\times$ lower variance	✓ Consistent ( Fig. 11 )

Table 4: Theoretical predictions vs. empirical findings (Uniform network,

\delta=0.2

T=10{,}000

Figure 12: Bias and standard deviation of the Hajék, HT, and DM estimators as a function of time-block length

\ell

, fixing

1/s=10

and

\delta=0.2

. Each curve corresponds to a different fractional parameter:

f_{r}\in\{0.25,0.5,1,1.5,2\}

for Hajék/HT and

f_{b}\in\{0,0.1,0.3,0.5\}

for DM. Shaded bands indicate

\pm 1

standard deviation.

Figure 13: MSE of the Hajék, HT, and DM estimators as a function of time-block length $\ell$ , fixing $1/s=10$ and $\delta=0.2$ . Each curve corresponds to a different fractional parameter: $f_{r}\in\{0.25,0.5,1,1.5,2\}$ for Hajék/HT and $f_{b}\in\{0,0.1,0.3,0.5\}$ for DM.

Figure 14: Bias and MSE of the DM estimator on the Hotel network as a function of

\ell

, for burn-in fractions

b/\ell\in\{0,0.1,0.3,0.5\}

. Each MSE curve has an interior minimum, and a small positive burn-in fraction substantially reduces bias with modest variance cost.

Figure 15: Bias and MSE of the HT estimator as a function of

\ell

, for recency values

r\in\{5,10,20,40,80,\ldots\}

with

\delta=0.2

. The HT estimator exhibits large MSE relative to Hajék (note the

y

-axis scale), and the optimal

r

depends on

\ell

Figure 16: Bias and MSE of the Hajék estimator as a function of

\ell

, for recency values

r

with

\delta=0.2

. The MSE scale (

\leq 0.02

) is far smaller than for HT ( Fig. 15 ), highlighting the efficiency advantage of the Hajék correction.

Causal Effect Estimation

Causal effect estimation measures how outcomes would change under interventions, using experiments or observational data and explicit assumptions.

See all 264 papers

Policy design in experiments with unknown interference
Nov 16, 2020Davide Viviano, Jess RudderCausal Effect Estimation Policy Optimization
Department of Economics, Harvard University, 1805 Cambridge St, Cambridge, MA 02138. · Department of Economics, Oregon State University, 1500 SW Jefferson Way, Corvallis, OR 97331.
Adaptive Policy Learning Under Unknown Network Interference
May 11, 2026Aidan Gleich, Eric Laber, Alexander VolfovskyThompson Sampling Multi-Armed Bandits
Department of Statistical Science Duke University
Spatiotemporal Proximal Causal Inference under Hidden Confounding and Interference
Aug 2, 2026Omar Faruque, Pavan Raj Ravi, Jianwu WangCausal Effect Estimation Proximal Causal Inference
University of Maryland, Baltimore County

Nov 16, 2020J/K move · Enter open · S save

Policy design in experiments with unknown interference

Davide Viviano, Jess Rudder

Department of Economics, Harvard University, 1805 Cambridge St, Cambridge, MA 02138. · Department of Economics, Oregon State University, 1500 SW Jefferson Way, Corvallis, OR 97331.

Causal Effect Estimation Policy Optimization Causal Inference Policy Learning

Open paper AskPDF arXiv

This paper studies experimental designs for estimation and inference on policies with spillover effects. Units are organized into a finite number of large clusters and interact in unknown ways within each cluster. First, we introduce a single-wave experiment that estimates the marginal effect of a change in treatment probabilities, taking spillover effects into account. Using the marginal effect, we propose a policy optimality test and improve the population's average outcome through marginal changes. Second, we design a multi-wave experiment to estimate optimal treatment rules, with strong regret guarantees. We provide an implementation in a large-scale field experiment studying climate adaptation.

Abstract

Causal Effect Estimation

Explore similar work

Policy design in experiments with unknown interference

Adaptive Policy Learning Under Unknown Network Interference

Spatiotemporal Proximal Causal Inference under Hidden Confounding and Interference