Organizations: School of Information Science and Engineering, Yunnan University, Kunming, China · Yunnan Key Laboratory of Intelligent Systems and Computing, Kunming, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
LLM agents require reliable step-level evaluation to compare candidate branches and allocate computation effectively. However, lightweight evaluation remains challenging. External verifiers introduce additional inference cost, while agent-produced confidence or self-evaluation scores can be miscalibrated, especially when candidates are generated by heterogeneous agents. We propose Generalized Layer-wise Intrinsic Distributional Evaluation (GLIDE) for LLM agents. GLIDE derives intrinsic step evidence from layer-wise residual coherence, which measures whether local residual updates consistently support the global residual change induced by a candidate step. It calibrates this evidence against the recent score distribution of the generating agent and converts it into a pessimistic reward that jointly accounts for absolute residual evidence and agent-relative standing. The reward provides a cross-agent value signal for MCTS branch selection, while normalized predictive uncertainty guides adaptive branching. Experiments on multi-hop reasoning, sequential decision making, and symbolic logic show that GLIDE improves task performance, step-level ranking quality, and computational efficiency without external verifiers or task-specific supervision.
Figures & tables
Figure 1: Score calibration mismatch across heterogeneous agents. Self-generated score distributions show that different LLM agents produce incompatible confidence scales on comparable inputs, making raw scores unreliable for cross-agent comparison.
Figure 2: Diagnostics of intrinsic stability in layer-wise evidence. Top : Transitions from successful trajectories (Green) concentrate in high-coherence/high-decisiveness regions, while transitions unique to failed trajectories (Red) are dispersed in the (Φcoh,Φdec) space. Bottom : Successful trajectories (Green) exhibit stable layer-wise residual evidence, whereas failed trajectories (Red) fluctuate and do not consistently progress toward a shared direction.
Figure 3: Overview of GLIDE
Table 4
Category
Method
P@1
MRR
NDCG@3
Efficiency
Hot
Web
G24
Avg
Hot
Web
G24
Avg
Hot
Web
G24
Avg
Time
Mem
Intrinsic
Perplexity
0.17
0.37
0.38
0.31
0.52
0.66
0.64
0.61
0.59
0.71
0.65
0.65
✓
✓
Energy
0.24
0.00
0.00
0.08
0.56
0.44
0.33
0.44
0.63
0.30
0.30
0.41
✓
✓
LN-Entropy
0.33
0.00
0.00
0.11
0.61
0.44
0.33
0.46
0.69
0.30
0.30
0.43
✓
✓
CoE-R
0.44
0.48
0.40
0.44
0.68
0.72
0.64
0.68
0.74
0.79
0.62
0.72
✓
✓
CoE-C
0.46
0.47
0.41
0.45
0.69
0.71
0.65
0.68
0.75
0.78
0.66
0.73
✓
✓
Table 4: Step-level preference quality of RGLIDE . We report ranking metrics (P@1, MRR, NDCG@3) on expanded child nodes from HotpotQA (Hot), WebShop (Web), and Game of 24 (G24), with Avg denoting the mean. Efficiency reports per-candidate scoring overhead; ✓ indicates negligible overhead. Hetero-Judge randomly selects scorers from the heterogeneous agent pool. Baselines are detailed in Appendix E.4.1 .
HotpotQA (Hot)
WebShop (Web)
Game of 24 (G24)
EM ( ↑ )
Cost (M) ( ↓ )
SR ( ↑ )
Score ( ↑ )
Cost (M) ( ↓ )
SR ( ↑ )
Cost (M) ( ↓ )
Pool Combo
Sym
Ours
Sym
Ours
Sym
Ours
Sym
Ours
Sym
Ours
Sym
Ours
Sym
Ours
Single: {L}
0.27
0.36
28.5
18.7
0.08
0.19
0.21
0.42
102.0
175.7
0.21
0.30
4.8
3.1
Single: {M}
0.32
0.44
11.1
14.9
0.35
0.35
0.65
0.67
52.9
48.0
0.15
0.23
25.5
11.5
Single: {Q}
0.30
0.49
53.3
12.1
0.39
0.46
0.66
0.71
54.4
20.2
0.14
0.14
9.6
4.6
Dual: {L,M}
0.52
0.53
24.4
12.0
0.37
0.45
0.73
0.77
82.5
44.0
0.31
0.40
6.4
6.5
Table 5: Controlled comparison across agent pools. Pools use L (Llama-3.1-8B-Instruct), M (Mistral-7B-Instruct-v0.3), and Q (Qwen2.5-7B-Instruct-1M). Performance uses EM, SR, and Score; Cost denotes total input/output tokens in millions. Teal marks the better result between SYMPHONY (Sym) and GLIDE .
Figure 4: Cross-task outcome separability of SISLE . We report average AUROC, FPR95, and AUPR across MGSM, GPQA, HumanEval, and LLMs from different families and scales. Higher AUROC/AUPR and lower FPR95 indicate stronger separation. Full results are in Appendix F.3 .
Variant
Hot
Web
G24
w/o Adap. Branch.
0.68
0.50
0.52
Raw SISLE
0.63
0.52
0.47
Calib. Judge
0.66
0.56
0.50
Rand. Agent
0.59
0.56
0.43
UCB Dispatch
0.65
0.55
0.46
Full
0.73
0.57
0.54
Table 6: Ablation Study (Full Pool; WebShop: SR).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Hyperparameter Sensitivity Analysis. Performance trends across three datasets (four metrics) under varying configurations. (a) Impact of the search branching bound nmax . The results show that performance remains generally stable across different branching bounds, with moderate gains obtained when allowing a slightly larger search space. (b) Impact of the sliding window capacity ∣W∣ . The sliding window balances statistical stability and adaptation to local distribution shifts. The results show a broad performance plateau around ∣W∣=50 , indicating that this default setting robustly covers typical context shifts without task-specific tuning.
Figure 6: Language-level MGSM outcome separability for Mistral-7B-Instruct-v0.3. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.
Figure 7: Language-level MGSM outcome separability for Gemma-3-12B-IT. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.
Figure 8: Language-level MGSM outcome separability for Phi-3-medium-128k-instruct. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.
Figure 9: Language-level MGSM outcome separability for InternLM2-Chat-20B. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.
Figure 10: Language-level MGSM outcome separability for Qwen2.5-32B-Instruct. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.
Figure 11: Language-level MGSM layer-subset analysis for Mistral-7B-Instruct-v0.3.
Figure 12: Language-level MGSM layer-subset analysis for Gemma-3-12B-IT.
Figure 13: Language-level MGSM layer-subset analysis for Phi-3-medium-128k-instruct.
Figure 14: Language-level MGSM layer-subset analysis for InternLM2-Chat-20B.
Figure 15: Language-level MGSM layer-subset analysis for Qwen2.5-32B-Instruct.
Figure 16: Prompt templates for cross-task outcome separability. We use task-specific prompts to standardize answer extraction across mathematical reasoning, scientific multiple-choice reasoning, and code generation.
Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.
Interactive large language model (LLM) agents operating via multi-turn dialogue and multi-step tool calling are increasingly used in production. Benchmarks for these agents must both reliably compare models and yield on-policy training data. Prior agentic benchmarks, such as tau-bench, tau^2-bench, and AppWorld, rely on fully deterministic backends, which are costly to build and iterate. We propose Proxy State-Based Evaluation, an LLM-driven simulation framework that preserves final state-based evaluation without a deterministic database. Specifically, a scenario specifies the user goal, user/system facts, expected final state, and expected agent behavior, and an LLM state tracker infers a structured proxy state from the full interaction trace. LLM judges then verify goal completion and detect tool/user hallucinations against scenario constraints. Empirically, our benchmark produces stable, model-differentiating rankings across model families and inference-time reasoning efforts, and its on-/off-policy rollouts provide supervision that transfers to unseen scenarios. Careful scenario specification yields near-zero simulator hallucination rates, as supported by ablation studies. The framework also supports sensitivity analyses over user personas. Human-LLM judge agreement exceeds 90%, indicating reliable automated evaluation. Overall, proxy state-based evaluation offers a practical, scalable alternative to deterministic agentic benchmarks for industrial LLM agents.
Inference-time compute scaling has emerged as a powerful technique for improving the reliability of large language model (LLM) agents, but existing methods apply compute uniformly: every decision step receives the same budget regardless of its difficulty. We introduce TrACE (Trajectorical Adaptive Compute via agrEement), a training-free controller that allocates LLM calls adaptively across agent timesteps by measuring inter-rollout action agreement. At each step, TrACE samples a small set of candidate next actions and measures how consistently the model commits to the same action. High agreement signals an easy decision; the controller commits immediately. Low agreement signals uncertainty; the controller samples additional rollouts up to a configurable cap before committing to the plurality action. No learned components, no external verifier, and no human labels are required. We evaluate TrACE against greedy decoding and fixed-budget self-consistency (SC-4, SC-8) on two benchmarks spanning single-step reasoning (GSM8K, n=50) and multi-step household navigation (MiniHouse, n=30), using a Qwen 2.5 3B Instruct model running on CPU. TrACE-4 matches SC-4 accuracy while using 33% fewer LLM calls on GSM8K and 39% fewer on MiniHouse. TrACE-8 matches SC-8 accuracy with 55% fewer calls on GSM8K and 65% fewer on MiniHouse. We further show that inter-rollout agreement is a reliable signal of step-level success, validating the core hypothesis that the model's own output consistency encodes difficulty information that can be exploited without training. TrACE is the first training-free, per-timestep adaptive-compute controller for LLM agents to be evaluated on multi-step sequential decision tasks.