cs.CLNov 16, 2026

When Should Decision Models Compare Candidates?

Authors: Fangzhi Li, Quan Shi, Wenting Yang, Yueqiao Chen, Jiayu Chang, Shuting Zhao, Zhaowei Xu, Yizhao Chen, 

Organizations: William & Mary · International School of Krakow · University of California, San Diego · University of California, Davis · 

Abstract

Language-model agents frequently select among candidate actions, such as tools and web elements. Should decision models score candidates independently or allow their scores to depend on other candidates? We investigate this question by comparing three approaches: Pointwise scoring, Setwise scoring, and Candidate Attention. Across five agent-decision benchmarks, Pointwise scoring remains competitive, while Setwise scoring performs better on some tool-selection and binary-decision tasks. Candidate Attention does not consistently improve upon Pointwise scoring, despite allowing cross-candidate interaction. Using two controlled synthetic tasks, we further examine whether models learn to use candidate-set information. With targeted relational training, Setwise models learn candidate dependencies and generalize to shifted inputs, while Candidate Attention shows weaker transfer. Our results show that access to other candidates does not guarantee effective comparison, and that its benefits depend on both task requirements and training.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MMDM: Measuring Massive Multitask Decision Making

    Nov 10, 2026Junhao Shen, Qing Zong, Yue Guo +3LLM Decision-MakingSystem One Decision Models

  2. Overwhelmed by Choice: Studying LLM Decision Making at Scale

    Sep 26, 2026Yu-Chi Lin, Aryan Seth, Anshul Aravind +4LLM EvaluationLLM Decision-Making

  3. TandemBench: Benchmarking Decision Models for Multi-Agent Collaboration

    Nov 4, 2026Baohang Li, Xiaocheng Feng, Bing QinAI Agent BenchmarksMulti-Agent Collaboration