GLIDE: Generalized Layer-wise Intrinsic Distributional Evaluation for Heterogeneous LLM Agents

Wei Zhu; Yiming Wang; Rui Wang; Lixing Yu; Kun Yue; Zhiwen Tang

cs.AISep 26, 2026

GLIDE: Generalized Layer-wise Intrinsic Distributional Evaluation for Heterogeneous LLM Agents

Authors: Wei Zhu, Yiming Wang, Rui Wang, Lixing Yu, Kun Yue, Zhiwen Tang

Organizations: School of Information Science and Engineering, Yunnan University, Kunming, China · Yunnan Key Laboratory of Intelligent Systems and Computing, Kunming, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China

Abstract

LLM agents require reliable step-level evaluation to compare candidate branches and allocate computation effectively. However, lightweight evaluation remains challenging. External verifiers introduce additional inference cost, while agent-produced confidence or self-evaluation scores can be miscalibrated, especially when candidates are generated by heterogeneous agents. We propose Generalized Layer-wise Intrinsic Distributional Evaluation (GLIDE) for LLM agents. GLIDE derives intrinsic step evidence from layer-wise residual coherence, which measures whether local residual updates consistently support the global residual change induced by a candidate step. It calibrates this evidence against the recent score distribution of the generating agent and converts it into a pessimistic reward that jointly accounts for absolute residual evidence and agent-relative standing. The reward provides a cross-agent value signal for MCTS branch selection, while normalized predictive uncertainty guides adaptive branching. Experiments on multi-hop reasoning, sequential decision making, and symbolic logic show that GLIDE improves task performance, step-level ranking quality, and computational efficiency without external verifiers or task-specific supervision.

Figures & tables

Figure 1: Score calibration mismatch across heterogeneous agents. Self-generated score distributions show that different LLM agents produce incompatible confidence scales on comparable inputs, making raw scores unreliable for cross-agent comparison.

Figure 2: Diagnostics of intrinsic stability in layer-wise evidence. Top : Transitions from successful trajectories (Green) concentrate in high-coherence/high-decisiveness regions, while transitions unique to failed trajectories (Red) are dispersed in the $(\Phi_{\mathrm{coh}},\Phi_{\mathrm{dec}})$ space. Bottom : Successful trajectories (Green) exhibit stable layer-wise residual evidence, whereas failed trajectories (Red) fluctuate and do not consistently progress toward a shared direction.

Table 4

Category	Method	P@1				MRR				NDCG@3				Efficiency
Category	Method	Hot	Web	G24	Avg	Hot	Web	G24	Avg	Hot	Web	G24	Avg	Time	Mem
Intrinsic	Perplexity	0.17	0.37	0.38	0.31	0.52	0.66	0.64	0.61	0.59	0.71	0.65	0.65	✓	✓
	Energy	0.24	0.00	0.00	0.08	0.56	0.44	0.33	0.44	0.63	0.30	0.30	0.41	✓	✓
	LN-Entropy	0.33	0.00	0.00	0.11	0.61	0.44	0.33	0.46	0.69	0.30	0.30	0.43	✓	✓
	CoE-R	0.44	0.48	0.40	0.44	0.68	0.72	0.64	0.68	0.74	0.79	0.62	0.72	✓	✓
	CoE-C	0.46	0.47	0.41	0.45	0.69	0.71	0.65	0.68	0.75	0.78	0.66	0.73	✓	✓

Table 4: Step-level preference quality of

R_{\mathrm{GLIDE}}

. We report ranking metrics (P@1, MRR, NDCG@3) on expanded child nodes from HotpotQA (Hot), WebShop (Web), and Game of 24 (G24), with Avg denoting the mean. Efficiency reports per-candidate scoring overhead; ✓ indicates negligible overhead. Hetero-Judge randomly selects scorers from the heterogeneous agent pool. Baselines are detailed in Appendix E.4.1 .

	HotpotQA (Hot)				WebShop (Web)						Game of 24 (G24)
	EM ( $\uparrow$ )		Cost (M) ( $\downarrow$ )		SR ( $\uparrow$ )		Score ( $\uparrow$ )		Cost (M) ( $\downarrow$ )		SR ( $\uparrow$ )		Cost (M) ( $\downarrow$ )
Pool Combo	Sym	Ours	Sym	Ours	Sym	Ours	Sym	Ours	Sym	Ours	Sym	Ours	Sym	Ours
Single: $\{L\}$	0.27	0.36	28.5	18.7	0.08	0.19	0.21	0.42	102.0	175.7	0.21	0.30	4.8	3.1
Single: $\{M\}$	0.32	0.44	11.1	14.9	0.35	0.35	0.65	0.67	52.9	48.0	0.15	0.23	25.5	11.5
Single: $\{Q\}$	0.30	0.49	53.3	12.1	0.39	0.46	0.66	0.71	54.4	20.2	0.14	0.14	9.6	4.6
Dual: $\{L,M\}$	0.52	0.53	24.4	12.0	0.37	0.45	0.73	0.77	82.5	44.0	0.31	0.40	6.4	6.5

Table 5: Controlled comparison across agent pools. Pools use

L

(Llama-3.1-8B-Instruct),

M

(Mistral-7B-Instruct-v0.3), and

Q

(Qwen2.5-7B-Instruct-1M). Performance uses EM, SR, and Score; Cost denotes total input/output tokens in millions. Teal marks the better result between SYMPHONY (Sym) and GLIDE .

Figure 4: Cross-task outcome separability of

S_{\mathrm{ISLE}}

. We report average AUROC, FPR95, and AUPR across MGSM, GPQA, HumanEval, and LLMs from different families and scales. Higher AUROC/AUPR and lower FPR95 indicate stronger separation. Full results are in Appendix F.3 .

Variant	Hot	Web	G24
w/o Adap. Branch.	0.68	0.50	0.52
Raw $S_{\mathrm{ISLE}}$	0.63	0.52	0.47
Calib. Judge	0.66	0.56	0.50
Rand. Agent	0.59	0.56	0.43
UCB Dispatch	0.65	0.55	0.46
Full	0.73	0.57	0.54

Table 6: Ablation Study (Full Pool; WebShop: SR).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Hyperparameter Sensitivity Analysis. Performance trends across three datasets (four metrics) under varying configurations. (a) Impact of the search branching bound

n_{\max}

. The results show that performance remains generally stable across different branching bounds, with moderate gains obtained when allowing a slightly larger search space. (b) Impact of the sliding window capacity

|\mathcal{W}|

. The sliding window balances statistical stability and adaptation to local distribution shifts. The results show a broad performance plateau around

|\mathcal{W}|=50

, indicating that this default setting robustly covers typical context shifts without task-specific tuning.

Figure 6: Language-level MGSM outcome separability for Mistral-7B-Instruct-v0.3. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.

Figure 7: Language-level MGSM outcome separability for Gemma-3-12B-IT. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.

Figure 8: Language-level MGSM outcome separability for Phi-3-medium-128k-instruct. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.

Figure 9: Language-level MGSM outcome separability for InternLM2-Chat-20B. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.

Figure 10: Language-level MGSM outcome separability for Qwen2.5-32B-Instruct. The figure shows the performance of different scoring methods across the 11 MGSM language versions. Higher AUROC/AUPR and lower FPR95 indicate better separation between successful and failed generations.

Figure 11: Language-level MGSM layer-subset analysis for Mistral-7B-Instruct-v0.3.

Figure 12: Language-level MGSM layer-subset analysis for Gemma-3-12B-IT.

Figure 13: Language-level MGSM layer-subset analysis for Phi-3-medium-128k-instruct.

Figure 14: Language-level MGSM layer-subset analysis for InternLM2-Chat-20B.

Figure 15: Language-level MGSM layer-subset analysis for Qwen2.5-32B-Instruct.

Panel of Figure 16 — Figure 16: Prompt templates for cross-task outcome separability. We use task-specific prompts to standardize answer extraction across mathematical reasoning, scientific multiple-choice reasoning, and code generation.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Jun 22, 2026Jeffrey FlyntLLM Agent Verification AI Agent Benchmarks
Independent Researcher
Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents
Feb 18, 2026Yun-Shiuan Chuang, Chaitanya Kulkarni, Alec Chiu +8LLM Agent Tool Generation LLM Agent Evaluation
PayPal AI
Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents
Apr 9, 2026Khushal SethiInference-Time Compute Allocation LLM Inference Efficiency
Stanford University

Jun 22, 2026J/K move · Enter open · S save

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Jeffrey Flynt

Independent Researcher

LLM Agent Verification AI Agent Benchmarks Tool-Use Evaluation LLM Grounding Tool-Augmented Language Model Agents LLM-as-a-Judge+8

Open paper AskPDF arXiv

Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.

Abstract

LLM Agent Evaluation

Explore similar work

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents

Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents