Organizations: Theremia, Paris · Paris Brain Institute (ICM), Sorbonne Université, Inserm, CNRS, AP-HP · Inria, Inserm, Université de Montpellier, France · Université Paris Cité, Inserm, Centre for Research in Epidemiology and Statistics (CRESS)
Treatment efficacy is traditionally demonstrated on the basis of a single primary outcome. However, clinical decision-making usually requires consideration of multiple outcomes, balancing expected benefits against potential risks. The relative value assigned to these outcomes varies substantially from one patient to another. Given a preference rule over outcome profiles, treatment effects and optimal policies can be defined and estimated. Such a rule is rarely available in practice: it must itself be estimated from pairwise comparisons of outcome profiles, which are costly to collect from clinical experts. We propose an active learning framework that selects which comparisons to query. Standard criteria maximize the information gained on the preference rule itself. We instead target the quantities of interest, and select the query that most reduces uncertainty on the treatment effect and on the optimal policy induced by the learned rule. Under a Gaussian process model of the preference rule, we derive a closed-form approximation of this criterion. On semi-synthetic data built from a Parkinson's disease cohort with 13 clinical outcomes, our criterion achieves lower treatment effect estimation error and lower policy regret than existing criteria at equal query budget.
Figures & tables
Figure 1: Overall workflow presented in the paper: CPTE-EIG asks the clinician only the comparisons that matter for the treatment decision.
Quantity
Parkinson’s disease instantiation
X
Full covariate. Includes comorbidities, PD subtype/staging, cognitive and psychiatric scales, anxiety, fatigue, biomarkers/genetics, ethnicity, race, education, family history, BMI.
X~=γ(X)
sex, age group ( <50 to 80+ , 5 -year bins), disease duration
Table 1: The quantities of the paper instantiated for Parkinson’s disease (PD), chosen in accordance with specialists. MDS-UPDRS ( Goetz et al., 2008 ) is the standard clinical scale for rating PD.
Figure 2: Mild treatment effect, per horizon h . Top: MSE of Tx(mt) against the oracle CPTE over 100 target patients from p⋆ . Bottom: regret of πt(x)=1{Tx(mt)>1/2} . Mean over 15 seeds, 95% bootstrap intervals. Lower is better.
Budget (number of comparisons)
Effect
Method
40
80
120
150
Tiny
Random
5.0 [4.0, 6.2]
2.9 [2.3, 3.5]
2.2 [1.8, 2.5]
2.5 [2.1, 3.0]
BALD
∗ 3.2 [2.7, 3.7]
1.3 [1.1, 1.4]
∗ 0.6 [0.5, 0.7]
∗ 0.3 [0.3, 0.4]
EPIG
3.3 [2.7, 3.9]
∗ 1.1 [1.0, 1.2]
0.7 [0.6, 0.8]
0.4 [0.3, 0.5]
CPTE-EIG
∗ 3.2 [2.8, 3.7]
1.3 [1.1, 1.4]
∗ 0.6 [0.5, 0.7]
0.4 [0.3, 0.5]
Mild
Random
12.4 [8.4, 17.0]
10.6 [7.6, 13.7]
6.0 [4.7, 7.3]
4.7 [3.3, 6.2]
Table 2: CPTE MSE ( ×103 ) against the number of comparisons, over 100 target patients from p⋆ (fixed across seeds and methods) and averaged over horizons: mean over 15 seeds and 95% percentile-bootstrap interval. Star ∗ for lowest mean (ties included); bold for methods with interval overlapping that of a ∗ method.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Tiny treatment effect. CPTE MSE (top) and policy regret (bottom) per horizon, as in Figure 2 .
Figure S2: Strong treatment effect. CPTE MSE (top) and policy regret (bottom) per horizon, as in Figure 2 .
Figure S3: Mild treatment effect. Accuracy (higher is better), Brier score and negative log-likelihood (lower is better) of the predicted answers on the 2,000 held-out comparisons. These measure the fitted preference rule, not the estimand.
Figure S4: CPTE MSE against the oracle, per horizon, for four rules approximating the expectation in Equation 22 – Equation 12 : Gauss–Hermite quadrature with n=10 and n=40 nodes, and Monte Carlo with 2,000 and 8,000 samples of Ψx .
Figure S5: Median wall-clock time per acquisition round (log scale), for the same four rules as Figure S4 .
Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We show the optimality of our instance-dependent term up to a constant depending on the logit bound, compared with a lower-bound characteristic. We evaluate our algorithms on real-world preference datasets and synthetic data observing improved sample efficiency compared to existing methods.
Yao Zhao, Kwang-Sung Jun
Department of Computer Science, University of Arizona, Tucson, USA.
Precision medicine aims to tailor therapeutic decisions to individual patient characteristics. This objective is commonly formalized through dynamic treatment regimes, which use statistical and machine learning methods to derive sequential decision rules adapted to evolving clinical information. In most existing formulations, these approaches produce a single optimal treatment at each stage, leading to a unique decision sequence. However, in many clinical settings, several treatment options may yield similar expected outcomes, and focusing on a single optimal policy may conceal meaningful alternatives. We extend the Q-learning framework for retrospective data by introducing a worst-value tolerance criterion controlled by a hyperparameter ε, which specifies the maximum acceptable deviation from the optimal expected value. Rather than identifying a single optimal policy, the proposed approach constructs sets of ε-optimal policies whose performance remains within a controlled neighborhood of the optimum. This formulation shifts Q-learning from a vector-valued representation to a matrix-valued one, allowing multiple admissible value functions to coexist during backward recursion. The approach yields families of near-equivalent treatment strategies and explicitly identifies regions of treatment indifference where several decisions achieve comparable outcomes. We illustrate the framework in two settings: a single-stage problem highlighting indifference regions around the decision boundary, and a multi-stage decision process based on a simulated oncology model describing tumor size and treatment toxicity dynamics.
Sophia Yazzourh, Erica E. M. Moodie
Department of Epidemiology and Biostatistics McGill University Montréal, Québec, Canada
Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.
Zhongman Du, Huiming Zhang, Haodong Zhu +1
Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University