cs.CLNov 4, 2026

Learning When Human Decisions Matter: Executor-Relative Consultation Value for LLM Agents

Authors: Shuli Jiang, Khushbu Pahwa, Yuting Zhang, Yi Zhang, Wei Xia, Stefano Soatto

Organizations: AWS Agentic AI

Abstract

Large language models increasingly automate complex tasks, but human decision capacity does not scale with their capabilities. In many applications, humans remain essential for providing private context, authority, or judgments unavailable to the agent. We formulate human decision capacity allocation: given multiple consultation opportunities and a limited budget, at which decisions should an LLM agent consult a human? The central challenge is estimating consultation value, which is executor-dependent because the same human input may benefit different downstream agents differently. We introduce Consultation Value Distillation (CVD), which estimates consultation values offline using Monte Carlo executor rollouts and distills these values into lightweight predictors for budgeted allocation, requiring no rollout-based value estimation at deployment. Across five batch and three sequential settings with two frontier executors, CVD improves allocation through joint and continuation-aware value learning, while independent consultation values remain competitive in several settings. CVD also broadly outperforms direct prompting and Value of Information where applicable, remains robust with substantially less rollout supervision, and enables nearly lossless approximate allocation at low search cost.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PORA: Learning When More Reasoning Pays Off for LLM Agents

    Oct 19, 2026Bo Yan, Qi Zhang, Zhen Tan +2Adaptive Reasoning in Language ModelsEfficient Language Model Reasoning

  2. Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents

    Feb 18, 2026Wenxuan Ding, Nicholas Tomlin, Greg DurrettLLM AgentsDecision-Making under Uncertainty

  3. JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

    Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent HarnessesLLM Agents