Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

Linhai Ma; Yan Wang; Lingfei Qian; Yupeng Cao; Victor Gutierrez Basulto

cs.AIOct 13, 2026

Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

Authors: Linhai Ma, Yan Wang, Lingfei Qian, Yupeng Cao, Victor Gutierrez Basulto

Organizations: The Fin AI, USA · Yale University, USA · Cardiff University, UK

Abstract

Tool-call judges evaluate argument values proposed by LLM agents. We find that they accept a correct value less often when their input holds no reference for it, that is, no text that says what the value should be. This happens when the value is a default value from the function schema that the user did not mention, when it has to be inferred, or when the text it comes from was cut from the judge input. We call this drop in acceptance the derivation gap. It appears across all seven judges we evaluate, including System One models CLM and Jev. The gap is not only a truncation artifact: correct default values have the largest gap on a benchmark with no truncation. A controlled source-relocation test shows that judges follow the reference in their input: they favor a visible incorrect value over a correct value whose source is hidden. We introduce DeRef (Derived Reference): a separate model reads the full history without seeing the candidate and writes a reference for the judge. Across three benchmarks, DeRef improves overall argument AUROC for all judges tested. On the controlled test, it raises AUROC from chance or below to between 98.40 and 99.59.

Figures & tables

Figure 1: A correct argument on the History-only path, from CFB. (a) The judge input keeps the first 1,000 characters of the search result, and the correct vehicle ID appears later. (b) DeRef asks a deriver to read the full history without the candidate and to write the expected value and its source. The reference goes into the same judge input. (c) Scores of the four judges, rounded to two decimals. With Plain input, every judge scores the value below 0.5 and below its own acceptance threshold

\tau

(Section 3.3 ); for Qwen3-8B, whose

\tau

on CFB is 0.002, the unrounded score is below that. With DeRef , every judge scores it above both.

	FCRB		CFB		FinTrace
Judge	Out of view	Default	Out of view	History-only	Out of view	No-source
CLM	8.15 [3.92, 12.07]	31.93 [24.36, 41.20]	30.76 [27.95, 34.02]	66.63 [59.42, 73.00]	25.95 [23.46, 28.22]	28.28 [25.34, 30.81]
Jev	8.11 [4.44, 11.74]	20.26 [12.68, 27.32]	22.67 [19.08, 27.71]	59.39 [54.22, 68.00]	27.78 [25.20, 29.87]	27.47 [24.54, 30.25]
Qwen3-4B	4.37 [0.37, 8.43]	21.26 [13.91, 29.63]	34.71 [32.87, 36.78]	76.81 [71.30, 81.93]	23.30 [20.91, 25.94]	23.29 [20.73, 26.35]
Qwen3-8B	0.63 [ $-$ 3.78, 4.35]	14.40 [4.69, 22.14]	33.30 [31.13, 35.65]	74.87 [68.09, 80.47]	24.04 [22.13, 26.22]	25.72 [23.59, 28.31]
Qwen3.8-27B	12.29 [8.19, 15.81]	36.03 [27.56, 44.48]	25.99 [21.10, 30.72]	57.08 [50.28, 63.87]	29.76 [27.44, 32.10]	31.68 [28.98, 34.22]

Table 1: Derivation gap

\mathrm{Gap}_{\tau}

(in points) with Plain input, with its 95% CI below. Out of view pools the four paths other than Visible . The second column of each dataset is its main path.

Figure 2: (a) Argument AUROC on FCRB-Relocate, with the source of the correct value in view (visible) or past the cut (buried). CLM is run at its default input length of 2,048 tokens and is not given the full history. (b) Acceptance at

\tau

of the 142 correct FCRB defaults whose DeRef reference says the default applies. Schema facts restates the default and lists related words in the request. Error bars show 95% CIs.

Judge and evidence	FCRB	CFB	FinTrace
CLM
Plain	72.51	63.41	83.36
ProvenanceGuard evidence ( She et al., 2026 )	75.17	69.62	84.75
RevisEval ( Zhang et al., 2025a )	83.26	81.78	86.07
DeRef (ours)	89.59	85.55	89.10
$\Delta$ vs. RevisEval	+6.33 [4.65, 7.98]	+3.77 [1.72, 5.90]	+3.03 [1.42, 4.79]

Table 2: Argument AUROC (in points) of four judges with each kind of evidence (Section 5.1 ). Bold marks the best input for each judge and benchmark.

\Delta

is DeRef minus RevisEval, with its 95% CI below. FCRB has no cut, so Full history equals Plain there. Table 16 adds DeRef -v and the gain over Plain.

Variant	AUROC	$\Delta$
DeRef	95.09	–
w/o full history	85.46	+9.63 [7.53, 11.86]
Qwen3-8B as deriver	88.56	+6.53 [4.70, 8.39]
Deriver thinking off	88.18	+6.91 [5.42, 8.47]
Parsed expected value only	90.90	+4.19 [3.15, 5.43]

Table 3: Ablation of DeRef on CFB with Qwen3.8-27B (argument AUROC in points). Each variant changes one part of DeRef . The judge reads the truncated input with thinking off throughout.

\Delta

is DeRef minus the variant, with its 95% paired bootstrap CI.

	Reference
Judge thinking	Without	With	$\Delta_{\mathrm{ref}}$
Off	85.45	96.01	+10.56 [8.82, 12.26]
On	90.27	94.74	+4.47 [2.74, 6.33]
$\Delta_{\mathrm{think}}$	+4.82 [3.18, 6.46]	$-$ 1.26 [ $-$ 1.86, $-$ 0.79]	–

Table 4: Reference availability and judge thinking on CFB with Qwen3.8-27B (argument AUROC in points). All four settings give the judge the full history. The two settings with a reference reuse the same references, written without the candidate and with deriver thinking on.

\Delta_{\mathrm{ref}}

is With minus Without within each row, and

\Delta_{\mathrm{think}}

is On minus Off within each column, with 95% paired bootstrap CIs.

Figure 3: Changing one value in the reference. Dots show the original reference, which names the correct value. Arrow heads show it with a wrong value from the same state.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	CFB		FinTrace
Judge	Full	Worst case	Full	Worst case
CLM	22.14	20.19 [18.13, 22.42]	5.74	1.87 [0.64, 2.90]
Jev	13.72	11.76 [9.79, 13.71]	3.99	0.89 [0.23, 1.30]
Qwen3-8B	16.34	14.05 [12.82, 16.28]	7.13	3.37 [2.46, 4.42]
Qwen3.8-27B	16.17	13.84 [11.72, 16.01]	2.94	1.15 [0.42, 2.13]
Llama-3.1-8B	16.98	15.37 [13.80, 16.47]	8.78	5.22 [4.29, 6.45]

Appendix

Table 5: Worst-case deletion of unconfirmed wrong labels. Full is the gain of DeRef over Plain in argument AUROC (points) on all wrong values. Worst case removes, for each judge, the 3% of CFB wrong values and 4% of FinTrace wrong values whose removal lowers that gain the most, and reports the gain that remains, with its 95% CI below. FCRB labels follow the BFCL checker and are not audited.

Judge	Plain	DeRef	$\Delta$
Relocate, buried (257 correct, 428 wrong)
CLM	43.73	98.40	+54.67 [51.22, 58.04]
Jev	45.00	98.70	+53.70 [49.74, 57.47]
Qwen3-8B	39.80	98.87	+59.07 [55.35, 62.76]
Qwen3.8-27B	49.03	99.59	+50.56 [46.98, 53.92]
Relocate, visible (257 correct, 428 wrong)

Appendix

Table 6: Argument AUROC (in points) on the two controlled sets.

\Delta

is DeRef minus Plain, with its 95% CI below. No difference is significantly negative.

		Out of view			Main path
Dataset	Judge	70%	80%	90%	70%	80%	90%
FCRB (Default)	CLM	5.85 [1.30, 10.89]	8.15 [3.92, 12.07]	6.20 [3.58, 9.51]	32.60 [24.82, 41.14]	31.93 [24.36, 41.20]	26.29 [19.87, 34.75]
	Jev	11.82 [7.35, 15.90]	8.11 [4.44, 11.74]	2.22 [ $-$ 0.75, 4.77]	32.99 [24.41, 41.88]	20.26 [12.68, 27.32]	9.41 [3.16, 14.70]
	Qwen3-4B	3.77 [ $-$ 0.43, 7.78]	4.37 [0.37, 8.43]	3.72 [0.75, 6.50]	25.82 [18.43, 34.44]	21.26 [13.91, 29.63]	15.64 [8.63, 22.33]
	Qwen3-8B	1.15 [ $-$ 2.98, 4.88]	0.63 [ $-$ 3.78, 4.35]	$-$ 1.00 [ $-$ 3.85, 1.77]	11.97 [3.59, 19.85]	14.40 [4.69, 22.14]	3.52 [ $-$ 1.60, 9.31]
	Qwen3.8-27B	16.93 [11.92, 21.19]	12.29 [8.19, 15.81]	6.07 [3.26, 8.78]	49.55 [40.32, 56.73]	36.03 [27.56, 44.48]	16.88 [10.16, 23.69]

Appendix

Table 7: Derivation gap

\mathrm{Gap}_{\tau}

(in points) when

\tau

is set to accept 70%, 80%, or 90% of correct arguments, with Plain input. The 80% columns repeat Table 1 . Out of view pools the four paths other than Visible ; the main path of each dataset is in parentheses. Llama-3.1-8B and Mistral-7B were run at 80% only, as additional rows of Table 1 , so the check covers the five judges run at all three rates. 95% CIs come from the same bootstrap, with

\tau

and

q

set again in every resample.

	Value	Correct / wrong	DeRef	EV only	$\Delta$
FCRB	Stated	1,577 / 620	94.69	94.93	$-$ 0.24 [ $-$ 0.89, 0.40]
	Not found	142 / 149	72.37	73.09	$-$ 0.72 [ $-$ 4.24, 3.13]
CFB	Stated	1,290 / 523	97.78	96.39	+1.39 [0.85, 2.05]
	Not found	348 / 470	82.78	60.41	+22.37 [16.99, 27.46]
FinTrace	Stated	3,384 / 1,763	95.99	97.11	$-$ 1.13 [ $-$ 1.46, $-$ 0.75]
	Not found	4,044 / 1,033	82.50	78.83	+3.67 [2.26, 5.33]

Appendix

Table 8: Keeping only the parsed expected value (EV only), split by whether the parser finds a value stated in the reference of the argument slot. Argument AUROC (in points) with Qwen3.8-27B.

\Delta

is DeRef minus EV only, with its 95% CI below. On CFB, the seven arguments of the one deriver prompt without an answer are left out.

	FCRB			CFB			FinTrace
Judge	Plain	DeRef	$\Delta$	Plain	DeRef	$\Delta$	Plain	DeRef	$\Delta$
CLM	66.19	85.84	+19.66 [17.10, 22.27]	52.56	73.47	+20.91 [17.38, 24.76]	80.34	81.55	+1.21 [0.00, 2.37]
Jev	83.46	88.12	+4.67 [3.15, 6.11]	56.56	78.67	+22.11 [18.18, 25.96]	87.50	90.06	+2.56 [1.88, 3.25]
Qwen3-8B	73.98	86.07	+12.09 [9.80, 14.19]	55.47	67.46	+11.99 [8.62, 15.51]	82.19	87.31	+5.12 [4.08, 6.21]
Qwen3.8-27B	85.43	88.72	+3.29 [1.74, 4.83]	68.78	85.98	+17.20 [13.84, 20.55]	86.69	88.08	+1.39 [0.63, 2.20]

Appendix

Table 9: Call AUROC (in points) with Plain input and with DeRef .

\Delta

is DeRef minus Plain, with its 95% CI below. Only the argument scores change with the input. For CLM on FinTrace the lower bound is 0.0005 points.

	FCRB	CFB	FinTrace
Mixed correct calls	325	197	1,107
Chance	50.45	48.77	48.73
CLM	62.77 [7.33, 17.62]	93.40 [40.97, 48.01]	75.07 [23.28, 29.28]
Jev	62.72 [7.49, 16.93]	85.25 [32.34, 40.93]	84.19 [32.85, 38.15]
Qwen3-4B	56.02 [2.02, 9.23]	91.53 [38.63, 46.33]	83.59 [32.23, 37.36]
Qwen3-8B	48.96 [ $-$ 5.02, 2.11]	90.36 [37.99, 45.27]	82.11 [30.98, 35.66]

Appendix

Table 10: Share (%) of correct calls whose lowest-scored argument is out of view, among correct calls with arguments both in and out of view, under Plain input. Chance is the share of out-of-view arguments in these calls. Below each share is the 95% CI of its difference from chance, in points. No share is significantly below chance.

Item	Setting
CLM	CLM-v0.1-8B ( Kwok et al., 2026 ) , one checkpoint file (SHA-256 b2b4a8c9…). We serve it with vLLM 0.30.0 in bf16. The score is the probability it gives to the answer “true” against “false”. We run it with the released serving settings, which truncate inputs to 2,048 tokens.
Jev	typesafe/jev-1.13-20260917 ( Almeida, 2026 ) , called through OpenRouter. In a length probe, it read inputs of up to 32,831 tokens by its own count and refused longer ones. A request holds only the state, the model id, and one question. We use the returned probability and set no sampling parameters.
Qwen judges	Qwen3-4B and 8B ( Yang et al., 2025 ) , and Qwen3.8-27B ( Qwen Team, 2026 ) , with 4.02B, 8.19B, and 27.78B parameters (26.90B in the text part of Qwen3.8-27B). We use one revision of each (1cfa9a7, b968826, and 1d4bf0f). They run in bf16 with transformers 5.18.0 and left padding. Thinking is off. The score is the softmax over the logits of “Yes” and “No” at the first output position. Where a judge runs with thinking on (Tables 4 and 15 ), it first generates its thinking, up to 8,192 tokens, after which we append the closing tag if it has not been written, and we then read the same two logits at the first position of the answer, without parsing the answer text. The two tokens hold less than half of the probability mass for at most 1.92% of inputs on any benchmark. Truncated inputs run in batches of 32. Full-history inputs run in batches sorted by length, at most 32 inputs and at most 65,536 padded tokens for Qwen3-8B and 32,768 for Qwen3.8-27B.
Other LLM judges	Llama-3.1-8B-Instruct ( Grattafiori et al., 2024 ) and Mistral-7B-Instruct-v0.3 ( Jiang et al., 2023 ) , revisions 0e9e39f and c170c70. We read them as the Qwen judges, in bf16 with left padding, with the same system prompt and question. Neither tokenizer has a padding token, so we pad with the end-of-sequence token.
Generation	Qwen3.8-27B with vLLM 0.30.0 in bf16, with thinking on at medium reasoning effort. Deriver prompts hold up to 32,768 tokens, and the oldest returns are dropped beyond that, which affects 190 of the 7,428 FinTrace deriver prompts. The deriver samples once per prompt with temperature 1.0, top-p 0.95, and top-k 20, the values recommended in the generation config of the model, and with seed 20260930. It writes at most 8,192 new tokens, and the reference is the text after the thinking. If no text follows the thinking, the judge gets the Plain input. This happens for one CFB prompt, which serves seven arguments, and no deriver prompt reaches the limit. ProvenanceGuard evidence and RevisEval use the same model and decoding settings. In the ablation, Qwen3-8B as deriver uses the values recommended for it, with temperature 0.6, and the deriver without thinking uses greedy decoding and at most 1,024 new tokens.
Hardware, software	Every job runs on one NVIDIA B200 GPU. Models run with Python 3.11.16, PyTorch 2.13.0 with CUDA 13.0, vLLM 0.30.0 ( Kwon et al., 2023 ) , and transformers 5.18.0. Analysis uses Python 3.12.11, NumPy 2.2.6, pandas 2.3.2, and SciPy 1.16.1.

Appendix

Table 11: Models and settings.

	Deriver (Qwen3.8-27B)					Judge input (Qwen3-8B tokens)
Benchmark	Prompts	Candidates / prompt	Input tokens	Output tokens	GPU s	Plain	DeRef	Full
FCRB	1,871	1.33	195 (365)	217 (968)	12	165 (352)	236 (444)	165 (352)
CFB	1,442	1.83	2,892 (29,327)	513 (4,763)	138	841 (2,031)	1,044 (2,084)	3,317 (29,673)
FinTrace	7,428	1.38	6,853 (32,651)	557 (6,351)	1,684	1,093 (2,084)	1,169 (2,084)	6,761 (32,646)

Appendix

Table 12: Cost of DeRef . Token counts give the mean and, in parentheses, the maximum. Deriver tokens use the tokenizer and chat template of Qwen3.8-27B, and judge tokens those of Qwen3-8B, so they include the template text. Output tokens include the thinking. GPU seconds cover generation without model loading. The run also held other prompts, so we split its time by prompt tokens.

Per input, batch size 1	FCRB	CFB	FinTrace
CLM, DeRef input (ms)	8.7	8.0	8.0
Qwen3-8B, DeRef input (ms)	34.7	66.9	57.6
Qwen3.8-27B, DeRef input (ms)	52.9	120.4	93.8
Qwen3.8-27B, direct judge with thinking, truncated input (s)	2.00	7.46	4.52
Qwen3.8-27B, direct judge with thinking, full history (s)	–	11.20	5.53
Deriver, one reference prompt (s)	2.44	7.27	6.34

Appendix

Table 13: Wall-clock time per input on one GPU in bf16, mean over 200 inputs drawn at random from each benchmark. The deriver row is the time to write one reference, which does not depend on the candidate and is shared by every candidate of that argument at that state.

	FCRB			CFB			FinTrace
Judge	Plain	DeRef	$\Delta$	Plain	DeRef	$\Delta$	Plain	DeRef	$\Delta$
Llama-3.1-8B	74.05	89.36	+15.31 [13.48, 17.02]	61.31	78.29	+16.98 [14.02, 19.97]	82.29	91.07	+8.78 [7.63, 10.12]
Mistral-7B	73.92	88.90	+14.97 [13.11, 17.12]	66.70	87.55	+20.84 [18.50, 23.54]	81.39	90.55	+9.16 [7.91, 10.40]

Appendix

Table 14: Argument AUROC (in points) of the two judges outside the Qwen family, with Plain input and with DeRef .

\Delta

is DeRef minus Plain, with its 95% CI below.

Qwen3.8-27B as	FCRB	CFB	FinTrace
Direct judge, thinking on, truncated input	89.79	84.17	88.67
Direct judge, thinking on, full history	–	90.27	89.70
Judge of the DeRef reference, thinking off	93.39	95.09	92.25
with the full history	–	96.01	92.52
Direct $-$ DeRef , truncated input	$-$ 3.60 [ $-$ 4.97, $-$ 2.35]	$-$ 10.92 [ $-$ 13.12, $-$ 8.76]	$-$ 3.58 [ $-$ 4.37, $-$ 2.83]
Direct $-$ DeRef , full history	–	$-$ 5.74 [ $-$ 7.40, $-$ 4.26]	$-$ 2.82 [ $-$ 3.57, $-$ 2.09]

Appendix

Table 15: Argument AUROC (in points) of Qwen3.8-27B judging directly with thinking on, against the same model reading the DeRef reference with thinking off. FCRB has no cut, so the full history equals the truncated input there.

\Delta

rows give the paired difference with its 95% CI below.

Judge and evidence	FCRB	CFB	FinTrace
CLM
Plain	72.51	63.41	83.36
ProvenanceGuard evidence ( She et al., 2026 )	75.17	69.62	84.75
RevisEval ( Zhang et al., 2025a )	83.26	81.78	86.07
DeRef (ours)	89.59	85.55	89.10
DeRef -v (ours)	86.18	88.93	85.88

Appendix

Table 16: Argument AUROC (in points) of four judges with each kind of evidence. ProvenanceGuard evidence and RevisEval are the prior methods (Section 5.1 ). DeRef -v lets the deriver see the candidate value (Section 5.3 ). Full history gives Jev and the LLM judges the uncut history (Appendix A ); FCRB has no cut, so there it equals Plain. Self-generated reference has the judge write its own reference from its own input (Section 5.1 ).

\Delta

rows give the difference of DeRef against Plain and against RevisEval, the stronger prior method in every cell, with its 95% CI below. Table 2 gives the main inputs, and Table 9 gives call AUROC.

Judge	Plain	Full history	DeRef	$\Delta$
FCRB , Default (185 correct)
CLM	31.93	–	3.80	$-$ 28.13 [ $-$ 36.67, $-$ 20.70]
Jev	20.26	–	10.28	$-$ 9.98 [ $-$ 15.63, $-$ 4.20]
Qwen3-8B	14.40	–	$-$ 3.93	$-$ 18.33 [ $-$ 25.50, $-$ 9.25]
Qwen3.8-27B	36.03	–	12.12	$-$ 23.91 [ $-$ 32.30, $-$ 15.69]
CFB , History-only (279 correct)

Appendix

Table 17: Derivation gap

\mathrm{Gap}_{\tau}

(in points) on the main path of each benchmark, with

\tau

set for each input. Full history gives Jev and the LLM judges the uncut history (Appendix A ), with its 95% CI below; CLM cannot read it, and FCRB has no cut.

\Delta

is DeRef minus Plain, with its 95% CI below. On FCRB, the acceptance at

\tau

of the 185 correct defaults is 52.43, 64.08, 65.95, and 50.54 with Plain input and 78.92, 76.03, 86.20, and 73.35 with DeRef , for the four judges in table order; for the 793 correct values in view it is 84.36, 84.34, 80.35, and 86.57 with Plain input and 82.72, 86.31, 82.27, and 85.47 with DeRef .

Input	CLM	Jev	Qwen3-8B	Qwen3.8-27B
CFB , History-only (279 correct, 59 wrong)
Plain (AUROC)	64.33	67.78	49.18	77.56
ProvenanceGuard evidence $-$ Plain	+5.01 [ $-$ 4.04, 13.34]	+7.81 [0.40, 16.07]	+17.64 [7.83, 26.98]	+4.25 [ $-$ 2.44, 11.69]
RevisEval $-$ Plain	+23.93 [16.90, 30.81]	+26.18 [17.28, 36.16]	+37.45 [27.23, 47.61]	+18.33 [10.81, 25.64]
Full history $-$ Plain	–	+12.80 [2.93, 23.17]	+25.95 [16.29, 34.90]	+11.45 [3.90, 19.93]
DeRef $-$ Plain	+27.16 [17.63, 36.74]	+21.83 [13.32, 30.54]	+43.69 [34.07, 53.29]	+16.87 [9.93, 23.04]

Appendix

Table 18: Argument AUROC (in points) on the two main paths of CFB. The first row of each block gives AUROC with Plain input, and the other rows give differences with their 95% CIs below. The wrong values of a path are those whose own value lies on that path. An argument with no item for an input keeps its Plain score. – marks inputs that we do not run with that judge.

Judge	Plain	Schema facts	DeRef
CLM	0.3718 / 0.6567	0.5092 / 0.6640	0.8924 / 0.7014
Jev	0.7300 / 0.8600	0.7300 / 0.8600	0.9600 / 0.9300
Qwen3-8B	0.0000 / 1.0000	0.0032 / 1.0000	1.0000 / 1.0000
Qwen3.8-27B	0.1824 / 0.6514	0.1824 / 0.4688	0.9903 / 0.9627

Appendix

Figure 5: A correct default on FCRB ( live_multiple_171-71-0 ). Each cell gives the judge’s score and its threshold

\tau

for that input (score /

\tau

). With DeRef , the score of Qwen3-8B equals its

\tau

	Plain		Original reference		Swapped reference
Judge	Correct	Wrong	Correct	Wrong	Correct	Wrong
CLM	0.7870	0.7632	0.8059	0.4389	0.3369	0.8281
Jev	0.9800	0.0600	0.9800	0.0100	0.4100	0.9700
Qwen3-8B	1.0000	1.0000	1.0000	0.0000	0.0000	1.0000
Qwen3.8-27B	0.3486	0.9579	0.9914	0.0003	0.0005	0.9890

Appendix

Figure 6: A swapped reference on FCRB ( live_multiple_809-176-0 ). The correct value is 17:00 and the wrong value is 15:00. Cells give scores.

Tool-Use Evaluation

Tool-use evaluation measures whether AI agents select, call, and rely on tools correctly, efficiently, and safely across realistic tasks.

See all 97 papers

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Aug 27, 2026Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian +1LLM-as-a-Judge LLM Agent Evaluation
ServiceNow AI Hyderabad, India
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
Aug 18, 2026Sher Badshah, Ali Emami, Hassan SajjadLLM-as-a-Judge Selective Prediction
Dalhousie University · New York University Abu Dhabi · Emory University

Aug 27, 2026J/K move · Enter open · S save

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian, Sai Harshitha Aluru

ServiceNow AI Hyderabad, India

LLM-as-a-Judge LLM Agent Evaluation Automated Agent Evaluation AI Agent Benchmarks Tool-Use Evaluation

Open paper AskPDF arXiv

LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.

Abstract

Tool-Use Evaluation

Explore similar work

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

LLM Judges Can Be Too Generous When There Is No Reference Answer

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees