The Revenue Cost of Inference in Online Assortment Optimization
Authors: Jierui Zuo, Hanzhang Qin
Organizations: Department of Management Science and Engineering, Tsinghua University, Beijing, China. · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.
A seller running an assortment experiment must earn revenue while collecting data to compare future offers. For a regular finite-dimensional parametric choice model and a finite assortment family, we characterize the oracle frontier between expected revenue loss and the largest local first-order variance constant across prespecified revenue comparisons. The oracle benchmark remains finite at zero cost exactly when revenue-maximizing assortments provide information in every target-gradient direction. When costly offers can supply the missing information, we derive the exact leading coefficient of the inverse-budget variance law and the limiting optimal spending shares. A pilot-based procedure reserves reference offers to preserve information and tightens the estimated budget constraint to account for uncertainty. We give conditions for the implemented design's variance criterion to converge to the stabilized oracle value at a fixed budget and reference share. Simulations with synthetic instances and Expedia-calibrated markets show that allocating offers across the full assortment family can improve the precision of revenue comparisons under a common opportunity-cost budget. Additional experiments assess pilot spending, budget protection, and sensitivity to choice-model misspecification.
Figures & tables
Figure 1: Learning customer preferences in Expedia Group’s experimentation process. Rankings A and B are compared using customer choices. Source: Authors’ schematic redrawing based on Figure 4 and the screening workflow in Woznica et al. (2026) .
Figure 2: Information-span characterization of the two local regimes. In the free regime, every target gradient lies in the information span of revenue-maximizing assortments. In the costly regime, at least one target gradient lies outside this span and requires positive-opportunity-cost traffic.
Figure 3: Two-track implementation. Base-policy and planned rounds are interleaved, with separate observation histories. The base policy updates only from its own observations. Pilot observations are used to tighten the opportunity cost budget and select a stabilized allocation, which is rounded before deployment. Contrast inference uses only post-pilot observations and conditions on the pilot-selected schedule.
Figure 4: The minimax frontier in exponent coordinates. Feasible operating points satisfy r≥max{1/2,2p} , and the segment p∈[1/4,1/2) , r=2p is pairwise rate-wise nondominated.
Design
Vθ⋆(q)
Cθ⋆(q)
Balanced singletons
1.1543
0.3276
Optimized singletons
0.6582
0.2747
Target-plus-reference
0.4109
0.1968
Full SDP
0.2864
0.06649
Table 1: Three-target synthetic illustration under a common budget ceiling. Oracle designs use the true parameter. Vθ⋆(q) is the largest first-order variance constant across the three targets. Cθ⋆(q) is expected opportunity cost per offer.
Figure 5: Oracle opportunity-cost–precision frontiers for all sixteen Expedia targets. Each curve represents one separately optimized contrast. Colors identify markets. Table EC.4 maps solid and dashed lines to the hotel contrasts in each panel. Both panels use a logarithmic budget axis. The free-inference panel uses a linear vertical scale, and the costly-inference panel uses a logarithmic vertical scale. Each curve equals one at the reference budget by normalization. The dotted line has slope −1 .
Market
Shared pilot
Full
Restricted
Equal pool
Optimized pool
D-optimal
4562
2,320.84
2.388 / 158.15
3.144 / 196.05
3.955 / 195.25
3.461 / 195.52
2.772 / 121.53
8192
2,065.57
2.266 / 129.23
2.917 / 141.48
3.274 / 140.98
2.893 / 141.35
2.813 / 153.77
9402
2,575.86
3.308 / 84.67
4.180 / 212.90
4.971 / 229.58
4.370 / 229.24
3.736 / 132.13
13870
2,021.69
2.697 / 72.62
4.506 / 123.35
3.864 / 121.57
3.386 / 122.05
3.468 / 126.20
Table 2: Joint common-ceiling results (20k/5k). Each method’s cell gives 103E∞ and cumulative post-pilot opportunity cost. Total cost adds the shared pilot. D-optimal uses true costs offline. 500 replications per cell.
Figure 6: Full joint SDP: reductions in post-pilot worst-target MAE under a common ceiling (left) and matched actual cost (right). Points show market-relative reductions. Bars show intervals propagated from simultaneous confidence bounds for target-level MAEs across the four markets. Matching and the common-ceiling D-optimal design use calibrated costs offline. Restricted matching uses 499 replications in markets 9402 and 13870 and 500 elsewhere. Other cells use 500.
Figure 7: Fixed total traffic of 25,000 at ηref . Total opportunity cost and mean post-pilot MAE are plotted against pilot size across four markets and eight prespecified targets. Points include all fallback runs. Error bars are 95% Monte Carlo intervals for the means. Pilot-size axes use a logarithmic scale, and lines connect the tested settings. The table reports observed certificate feasibility, counting each shared pilot once per market. Each setting uses 500 unique pilot replications per market.
Figure EC.1 : Schematic oracle frontiers between revenue and precision. The vertical axis measures the largest local variance constant across the target contrasts under the best allocation for each budget. Lower values indicate greater precision. Free instances have a finite zero-budget intercept. When first-order inference is feasible at a positive budget, costly instances have Fθ(0)=+∞ and Fθ(η)=Θ(1/η) as η↓0 .
Test log loss
Market
Anchor
Retained
Calib.
Valid.
Test
Hotel-specific
Shared
Difference
4562
53494
2501
1539
497
465
0.5820
0.5879
0.0059
8192
116942
4691
2870
916
905
0.8759
0.8812
0.0053
9402
35679
1976
1172
409
395
0.6366
0.6462
0.0095
13870
106230
1720
1055
323
342
0.7132
0.7156
0.0024
Table EC.1: Historical searches and mean test log loss. The anchor is the selected hotel with the highest normalized calibration median price. The loss difference is the shared-attraction loss minus the hotel-specific loss on the displayed test menus.
Post-pilot estimation
Empirical all-planned refit
Market
Reduction
95% interval
Holm
Reduction
95% interval
Holm
4562
22.19
[18.43, 25.58]
4/4
7.29
[3.56, 10.59]
0/4
8192
24.96
[21.37, 28.21]
4/4
12.72
[8.61, 16.31]
1/4
9402
28.18
[24.65, 31.46]
4/4
14.28
[10.66, 17.65]
3/4
13870
21.85
[18.17, 25.26]
4/4
11.19
[7.35, 14.65]
3/4
All sixteen targets
24.29
[22.48, 25.86]
16/16
11.37
[9.47, 13.03]
7/16
Table EC.2: Tightened pilot SDP relative to D-optimal at matched post-pilot opportunity cost ( λ=1 ). Entries are equal-weight mean target-relative MAE reductions, in percent. 95% intervals use joint replication-block resampling within fixed markets. Holm counts adjust sixteen comparisons separately for each sample convention. Both methods share the pilot and each uses 5,000 post-pilot observations.
λ
Pilot-best +reference
Pilot-matched mixture
D-optimal (pilot-tightened)
D-optimal 95% interval
0
17.35
18.10
12.26
[10.38, 13.98]
0.1
17.25
18.02
12.22
[10.34, 13.94]
1
16.45
17.41
11.97
[10.10, 13.69]
Table EC.3: Sensitivity to inference regularization with allocations and samples fixed. Entries are mean target-relative reductions in all-planned MAE, in percent. The D-optimal comparator uses the pilot-tightened budget. Pilot fits retain λ=1 . The last column gives the stratified paired bootstrap interval for the D-optimal comparison.
Panel A. Zero-floor frontier
Market
Hotel
Regime
Line
F(0)
104Lθ
Optimal paid offer
4562
61529
Free
Solid
0.035327
—
—
4562
79740
Free
Dashed
0.033463
—
—
4562
21155
Costly
Dashed
∞
0.313835
{53494,79740,21155}
4562
14536
Costly
Solid
∞
1.356274
{53494,79740,14536}
8192
77089
Costly
Solid
∞
1.017735
{116942,77089,104517}
Table EC.4: Oracle targets and the cost of reference stabilization. Panel A gives the line mapping, exact leading coefficient and paid offer for the zero-floor frontier. Panel B gives envelopes of certified τϵ⋆/Fθ intervals across four targets per regime. Here x=η/ηref , and the gap is the largest relative primal–dual gap across both oracle values in each feasible row. The constraint is infeasible when x<ϵ .
Contract
Comparator
Post gain
95% interval
All gain
95% interval
Ceiling
Restricted
26.84
[8.98, 41.14]
10.34
[-10.96, 27.56]
Ceiling
Equal pool
33.51
[17.12, 46.61]
10.85
[-9.97, 27.76]
Ceiling
Optimized pool
24.34
[6.14, 38.99]
10.03
[-11.74, 27.51]
Ceiling
D-optimal
16.74
[-3.58, 33.06]
11.30
[-9.74, 28.30]
Match
Restricted a
41.98
[27.71, 53.36]
8.04
[-13.63, 25.58]
Match
Equal pool
26.96
[8.64, 41.48]
7.08
[-14.89, 24.84]
Table EC.5: Worst-target MAE reductions for Full relative to each comparator, averaged across the four markets, with propagated 95% intervals (percent). a Restricted matching uses 1,998 available market-replications. Other comparisons use 2,000. The spend cap equals Full’s actual post-pilot opportunity cost.
Comparator
Post gain
95% interval
All gain
95% interval
Restricted
30.25
[13.36, 43.68]
18.98
[0.12, 34.28]
Optimized pool
14.74
[-5.30, 30.84]
12.05
[-8.26, 28.49]
D-optimal
7.10
[-14.93, 24.76]
5.99
[-15.61, 23.54]
Table EC.6: Independent 5k/20k validation under correct MNL specification. Worst-target MAE reductions for Full relative to each comparator, averaged across the four markets (percent). All methods share the box certificate and estimated costs. Actual costs can differ across methods. Each market uses 500 fresh replications.
Full post-pilot
Full all-planned
Direct
Total cost
δ
MAE / bias a
Coverage (%)
MAE / bias a
Coverage (%)
MAE / bias a
Coverage (%)
Full
Direct
Panel A. 20k/5k, Lipschitz certificate
0
2.665 / 0.937
95.6
1.803 / 0.338
94.6
4.570 / 0.424
95.0
2,357.16
1,528.43
0.5
2.771 / 1.099
95.8
1.833 / 0.460
92.8
4.732 / 0.380
94.6
2,372.63
1,546.08
1.5
4.371 / 3.834
84.0
3.032 / 2.614
56.0
5.091 / 0.361
94.0
2,655.79
1,852.29
Panel B. 5k/20k, box certificate
Table EC.7: Misspecification at 25,000 total offers. In each market, E∞ takes the maximum across targets of the mean absolute error over 500 replications. Bias is the largest absolute target mean error. The table averages E∞ , bias, and total opportunity cost over the four markets. Coverage is the minimum across markets of empirical simultaneous Bonferroni Wald coverage. Full refits share the executed schedule and cost.
Comparator
Common ceiling
Matched cost
Independent 5k/20k
Restricted
0.061[0.015,0.107]
0.430[0.315,0.551]
0.013[−0.013,0.039]
Equal pool
0.057[0.006,0.110]
0.058[0.010,0.107]
—
Optimized pool
0.034[−0.012,0.079]
0.059[0.010,0.108]
0.024[−0.003,0.050]
D-optimal
−0.028[−0.070,0.012]
−0.010[−0.052,0.030]
−0.005[−0.031,0.022]
Table EC.8: Selection loss differences (comparator minus Full) with paired 95% intervals, scaled by 103 . Intervals are exploratory and unadjusted across comparisons. We use equal market weights and 20,000 within-market pair resamples. Offline true-cost matching has 1,998 Restricted pairs. Other cells have 2,000.
Department of Biostatistics, Harvard T.H. Chan School of Public Health · Leonard N. Stern School of Business, New York University · Department of Biostatistics & Bioinformatics, Duke University