Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships

Yuanhao Pu; Defu Lian; Enhong Chen

cs.LGMar 8, 2026

Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships

Authors: Yuanhao Pu, Defu Lian, Enhong Chen

Organizations: School of Artificial Intelligence & Data Science, University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence, China · School of Computer Science & Technology, University of Science and Technology of China

Abstract

The Consistency property between surrogate losses and evaluation metrics has been extensively studied to ensure that minimizing a loss leads to metric optimality. However, the direct relationship between different evaluation metrics remains significantly underexplored. This theoretical gap results in the "Metric Mismatch" frequently observed in industrial applications, where gains in offline validation metrics fail to translate into online performance. To bridge this disconnection, this paper proposes a unified theoretical framework designed to quantify the relationships between metrics. We categorize metrics into different classes to facilitate a comparative analysis across different mathematical forms and interrogates these relationships through Bayes-Optimal Set and Regret Transfer. Through this framework, we provide a new perspective on identifying the structural asymmetry in regret transfer, enabling the design of evaluation systems that are theoretically guaranteed to align offline improvements with online objectives.

Figures & tables

Panel of Table 1 — Table 1: Cross-Analysis of Metric Relationships: From Consistency to Mismatch.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer
Aug 2, 2026Yuyang ShenSurrogate Modeling Decision-Focused Learning
The Chinese University of Hong Kong, Shenzhen
Parsimonious Learning-Augmented Online Metric Matching
May 26, 2026Yongho Shin, Phanu VajanopathLearning-Augmented Algorithms
Institute of Computer Science, University of Wrocław, Wrocław, Poland.
Evaluation Metrics as Averaged Outcomes of Fair Gambles
Jan 25, 2024Rabanus Derr, Robert C. Williamson
University of Tübingen and Tübingen AI Center Tübingen, 72076, Germany

Aug 2, 2026J/K move · Enter open · S save

When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer

Yuyang Shen

The Chinese University of Hong Kong, Shenzhen

Surrogate Modeling Decision-Focused Learning Gradient Descent Dynamics Gradient Alignment

Open paper AskPDF arXiv

A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward. Here, a trajectory is a training instance that induces a surrogate loss whose reduction might not track the model's decision utility update. Theoretically, we ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates. To formalize this, we first fix a checkpoint and a restricted update space, and define the reductions in population surrogate risk and decision risk induced by a trajectory as its learnability and decision utility, respectively. On this basis, our theory yields four main results. First, a one-step transfer bound separates their discrepancy into first-order gradient misalignment after nonnegative calibration and second-order curvature; and a pathwise extension accumulates the same terms over repeated updates. Second, when the accessible surrogate gradient is nonzero, universal first-order transfer over every accessible direction holds exactly when the accessible surrogate and decision gradients are positively collinear. Third, the calibration gap bounds the decision regret of learnability-based trajectory selection, while a candidate-difference refinement tightens this guarantee by retaining only directions that affect pairwise rankings. Finally, we establish an approximation--calibration trade-off across nested update spaces. Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.

Abstract

Automated Evaluation

Explore similar work

When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer

Parsimonious Learning-Augmented Online Metric Matching

Evaluation Metrics as Averaged Outcomes of Fair Gambles