What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels

Jiawei He; Mengyu Shi; Jie jia; Xikai Yang; Dong Sun

cs.AIAug 24, 2026

What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels

Authors: Jiawei He, Mengyu Shi, Jie jia, Xikai Yang, Dong Sun

Organizations: Amap, Alibaba Group, China

Abstract

Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SCAE . An episode is rolled out once to obtain $(\tau_{1:T},Y)$ . Prefix replay reconstructs $\tau_{1:t}$ by rewriting the rollout log at item granularity, which requires no modification to the agent. An entropy probe at each estimation point decides which identification strategy applies: where the policy retains randomness, $v_{t}$ is estimated observationally by fork-resampling (Proposition 1 ); where it is effectively deterministic, an alternative action is substituted and actually executed (Assumption 4 ). The single fitted object $\widehat{v}_{t}$ then supports three readings. We report what each delivered rather than what it promised: the increments do not localize at feasible cost (§ 4.5 ), the trajectory yields a task-level rather than step-level uncertainty (§ 4.5 ), and the endpoint $v_{0}$ does not forecast the outcome (leave-one-repository-out AUROC 0.481). The decomposition of § 3.8 is validated on a synthetic generating process, not on these traces. In the figure, shaded boxes are data and plain boxes are pipeline stages; the two branches below the entropy probe are the two cases of equation 14 . Algorithm 1 in Appendix B.1 states the procedure.

Panel of Figure 2 — Figure 2: Three conditioning sets and their causal status . The conditioning set, not the scorer, determines whether attribution is causal. Latent task difficulty $Z$ (dashed) influences both behaviour and outcome, but conditioning on the prefix $\tau_{1:t-1}$ blocks it, because the policy observes nothing that is not logged and decoding noise is exogenous. Conditioning additionally on $A_{t+1}$ and $Y$ , which is what full-trace judging does, conditions on descendants of $A_{t}$ and opens a biasing path. The predicted signature is attribution displaced toward later steps. We test it in § 4.6 by a controlled manipulation of the judge’s information set, which is the one attribution result here that requires no causal reference.

Object model	cov.	top-1	top-3	top-10	NLL	ECE	$\exp(H)$
Routed by predicted tool	0.619	0.150	0.260	0.484	2.76	0.0123	12.8
Factored, shared object model	0.645	0.109	0.241	0.432	3.41	0.0139	39.9
Paired difference	—	+0.041	+0.018	+0.052	—	—	—
95% CI	—	[+.034, +.047]	[+.009, +.026]	[+.047, +.058]	—	—	—

Table 1: Predicting the complete next action

(\mathrm{tool},\mathrm{obj})

over 7616 decision points from 488 episodes. Both models share the same tool component and the same denominator: every decision point counts, and points whose realized action falls outside the candidate set are scored as misses rather than dropped. top-

k

are closed-form expectations over random tie-breaking. NLL is over covered points only, so it measures sharpness rather than coverage.

\exp(H)

is the median effective candidate count. Routing the construction of the candidate set on the predicted next tool—reads take paths, searches take free-text queries, handler tools take no object—is what the dependence

I=0.159{}

nats (

p=5\times 10^{-4}{}

) between tool and object type requires; it improves every top-

k

and tightens the candidate set

3.1\times

Stratum (observable before predicting)	$n$	top-3	ECE
Current step is a search	228	0.469	0.0125
Current step is a read	206	0.160	0.0247
Current step lists a directory	35	0.057	0.0278
Predicted $\exp(H)\leq 5$	154	0.351	0.0762
Predicted $\exp(H)\in(5,15]$	198	0.409	0.0168
Predicted $\exp(H)>15$	157	0.159	0.0157

Table 2: When the next-action distribution can be trusted. Only strata observable before the prediction are listed, because a stratum defined by the realized answer is a conditional accuracy and not a rule: conditioning on coverage gives top-3

=0.611

at ECE

=0.0103

, which is not actionable and which we therefore exclude. Two readings are practical. Prediction is trustworthy immediately after a search and untrustworthy after a read, an eight-fold spread. And low entropy is not sufficient for trust: the

\exp(H)\leq 5

stratum is the worst calibrated, because a small candidate set signals coverage risk rather than confidence.

Next-object predictor	cov.	top-3	95% CI	top-10	$\exp(H)$
Repository structure
Repository frequency (marginal)	0.349	0.072	[0.047, 0.099]	0.163	97.62
Code dependency graph, strengthened	0.677	0.058	[0.039, 0.078]	0.172	1719.13
Visit history
Visit-history frequency	0.000	0.000	[0.000, 0.000]	0.000	4.84
Execution provenance

Table 3: The RQ1 predictor suite on the 341 first-visit decision points, where the realized object has never been visited before. This is the discriminating regime: visit history covers none of these steps by construction, so a predictor cannot score by restating the agent’s own past, which is why the visit-history row sits at zero coverage. All rows share one denominator—points whose realized object falls outside the candidate set are scored as misses rather than dropped—and top-

k

are closed-form expectations over random tie-breaking. The code-graph row is strengthened relative to the obvious implementation: it seeds a multi-source proximity ranking from the entire visited set rather than from the last step alone, and it still loses to every provenance variant listed, including the uninformative uniform one. The paired contrast quoted in § 4.2 uses the frequency-weighted row rather than the best row, so it understates the gap. Higher is better for coverage, top-3 and top-10; lower is better for the effective candidate count

\exp(H)

. Best value per column in bold.

Figure 3: Execution provenance versus code structure on first-visit steps . The structure that predicts an agent’s next object is its own execution provenance, not the code dependency graph. (a) On the 67.0% of steps whose object has never been visited—where visit history is empty by construction—a predictor reading only the most recent tool output reaches top-3

=0.326{}

, against

0.058{}

for a code graph strengthened to seed a multi-source proximity ranking from the entire visited set, and

0.072{}

for repository frequency. Bars are closed-form expectations over random tie-breaking; whiskers are episode-level bootstrap 95% CIs. (b) Predictability is heterogeneous, and bar width is the share of all 7616 steps: handler steps are nearly determined, path steps are moderately predictable, and steps taking a novel free-text search query are not. A single aggregate number would hide this.

Figure 4: What the dependency graph does and does not describe . The graph describes where an agent works, not how it moves. (a) Against a within-episode permutation null—which preserves the multiset of visited files and destroys only their order—consecutive steps are no closer in the graph than chance (one-hop lift

0.94{}

with self-repeats removed, three-hop

1.02{}

, same-community

1.01{}

). The only quantity above chance is distance zero, i.e. immediate self-repetition, which is a temporal effect (§ 4.4 ). (b) The same graph separates the visited set from random same-size sets by

30.9{}\times

in internal edge density. The two panels are consistent: inside a region of density

0.269{}

an arbitrary pair is already likely to be one hop apart, and the permutation null removes exactly that region effect.

Figure 5: The level at which execution uncertainty resides . Execution uncertainty is a property of the task, not of the step. (a) Probes for structure in the replay variance

U^{\mathrm{exec}}

, all expressed as a share of variance explained. Instance identity explains

0.640{}

against a permutation null of

0.196{}

(

p=5\times 10^{-4}{}

); step position and tool type explain nothing. (b) The mechanical mean–variance relation that had to be removed before (a) is credible:

U^{\mathrm{exec}}

collapses to zero in the top quintiles of

v_{t}

, because variance is constrained by the mean. Panel (a) is computed on residuals after binning on

v_{t}

. Given the absence of step-level structure, the failure of step-level effects to survive FDR correction is expected rather than remediable by a denser scan.

Figure 6: Collider isolation in full-trace judging . Conditioning on descendants of the scored step displaces the verdict later, and the manipulation identifies this by construction rather than by inference. (a) Each line is one episode. Judge model and rubric are fixed; the only difference is whether steps downstream of the scored step are visible. The verdict moves later in 59 of 70 episodes (10 ties), by

+0.537{}

in mean normalized position. Because the manipulation is of the judge’s information set rather than of the world, this requires no causal reference—which matters here, since none can be certified (§ 4.5 ). (b) What the full-trace judge selects on: steps that operate on or reveal a gold file, at

0.900{}

against the correct baseline of

0.413{}

, and biased late within that class against a closed-form uniform null. That is semantic relevance—cheap and reproducible, but not demonstrably causal contribution.

Zero-LLM structural rule	agree	chance	diff	sig.
First step revealing a gold file	0.043	0.053	-0.010	—
First step operating on a gold file	0.100	0.053	+0.047	—
Last gold-provenance step	0.243	0.053	+0.190	✓
Last step of the trace (position only)	0.171	0.053	+0.118	✓
Step revealing the most gold files	0.043	0.053	-0.010	—

Table 4: Can a rule with no language model reproduce the full-trace judge’s choice? Agreement is exact same-step agreement over 70 episodes; chance is the closed-form expectation

1/T_{i}

for a rule selecting uniformly among steps. The judge’s class of choice is almost fully explained—it selects gold-provenance steps at 0.900 against a 0.413 baseline, and is biased late within that class (

0.746{}

against a uniform null of

0.5

). Its exact choice is not: with

8.7{}

gold-provenance steps per episode the within-class chance rate is

0.139{}

, so the best rule is only

1.7\times

better than guessing inside the class. We report this residual rather than presenting the judge as a lookup table.

	Collider displacement				Gold-prov. enrichment
Judge	mean	95% CI	later	sign $p$	rate	diff CI
gpt-5.5 (original)	+0.533	[+.399, +.656]	19/23	$4{\times}10^{-6}$	0.826 ∗	[+.241, +.500]
qwen3.6-plus (general)	+0.434	[+.299, +.557]	18/23	$8{\times}10^{-6}$	0.565	[-.068, +.297]
qwen3-coder-plus (code)	+0.394	[+.251, +.551]	15/23	$6{\times}10^{-5}$	0.826 ∗	[+.222, +.514]

Table 5: The same manipulation across 3 judges on their 23 common episodes. Displacement—the effect of making steps downstream of the scored step visible—appears in every judge and shows no detectable between-judge heterogeneity; the spread of

0.14{}

across judges is small against per-judge intervals of roughly

\pm 0.13

, which is what supports the claim, since a

p

0.181

on 23 episodes means only that no difference was detected. Enrichment behaves oppositely: it reaches significance in 2 of 3 judges ( ∗ ) and its heterogeneity is significant. The baseline for the enrichment column is

0.443{}

, the share of steps that touch or reveal a gold file. A critique of the mechanism therefore transfers across scorers; a critique of what they score does not.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$q$	natural-language issue text of an episode
$G=(V,E)$	repository graph over files, modules and symbols
$F^{\star}$	gold target set: files modified by the reference patch
$\hat{F}$	file set predicted by the agent
$Y$	task-level outcome $\mathbb{1}[F^{\star}\subseteq\hat{F}]$
$Y_{f}$ , $Y_{\mathrm{rec}}$	per-file indicator; recall $\|F^{\star}\cap\hat{F}\|/\|F^{\star}\|$

Appendix

Table 6: Notation used throughout the paper. Symbols are introduced in § 3 and reused without redefinition in § 4 .

Practice	Conditions on	Consequence
Full-trace LLM judge	prefix $+$ successors $+$ outcome	Successors are descendants of $A_{t}$ , and conditioning on a descendant opens a biasing path. Predicted signature: blame displaced toward later steps.
Process reward models	nominally the prefix	Labels back-filled from completed trajectories reintroduce future information.
Ablation with restart	nothing	Resamples all noise, so it estimates a marginal interventional contrast and discards the realized history.
Data-flow “dead step” analysis	syntactic dependencies	Conflates data-flow reachability with causal effect on the outcome (§ A.7 ).

Appendix

Table 7: Each row misestimates Eq. equation 1 for a distinct reason, and none of the four reasons is unmeasured confounding. Only the first row yields a signature testable without a causal reference, which is why it is the attribution result we foreground in § 4.6 .

Level	Expression	Meaning for localization	Typical practice
I	$P(\hat{F}\mid q)$	association between query and result	retrieval recall@ $k$
II	$P(\hat{F}\mid do(a^{\prime}))$	alter a step and restart	tool-ablation studies
II.5	$P(\hat{F}\mid\tau_{1:t-1},do(a^{\prime}))$	fix realized history, alter one step	SCAE (continuation)
III	$P(\hat{F}_{a^{\prime}}\mid\tau_{\mathrm{obs}})$	fix realized noise, alter one step	SCAE (intervened step)

Appendix

Table 8: Levels II and II.5 differ in whether the episode’s own history is retained. Most agent ablation studies operate at Level II while being described in counterfactual language, which is the confusion the distinction is meant to prevent. § A.4 states where the boundary falls for SCAE.

Episode	Entropy (nats) by normalized position	Pattern
django-10097 (16 steps)	$0.12{:}\,0\rightarrow 0.38{:}\,1.04\rightarrow 0.62{:}\,1.04$	det.–stoch.–stoch.
astropy-14369 (29 steps)	$0.10{:}\,0\rightarrow 0.31{:}\,0.56\rightarrow 0.52{:}\,0.56\rightarrow 0.76{:}\,0$	det.–stoch.–det.
astropy-12907 (8 steps)	$0.25{:}\,0.45\rightarrow 0.50{:}\,0\rightarrow 0.75{:}\,0$	stoch.–det.

Appendix

Table 9: Action entropy at all 10 probed decision points across 3 episodes. Overlap is present at some steps and absent at others within the same episode , and the three episodes realize three different patterns. This is what forces the per-step branch selection of Eq. equation 14 : neither a globally observational nor a globally interventional estimator is correct. Entropies fall on a coarse grid because each point is probed with

4

–

6

replays.

Figure 7: Overlap is a local property, not a global one . Each marker is one probed decision point, positioned by its normalized position in the episode and coloured by episode. Points on the zero line admit no observational identification and are routed to the intervention branch of Eq. equation 14 ; points above it are estimated observationally under Proposition 1 . 50% of points lie above zero, and the rank correlation with position is only +0.072, so overlap is not a monotone function of trajectory depth. We report the sample size plainly: 10 points establish that overlap varies within episodes, which is all the estimator requires, but not the distribution of that variation, and the near-zero rank correlation should not be read as evidence of no relationship.

Repository	$n$	Localization success
django	$230$	$93.5\%$
sympy	$75$	$94.7\%$
sphinx-doc	$44$	$88.6\%$
matplotlib	$34$	$91.2\%$
scikit-learn	$32$	$100.0\%$
astropy	$22$	$100.0\%$

Appendix

Table 10: Localization success by repository under the criterion

F^{\star}\subseteq\hat{F}

. The corpus is dominated by django at

46\%

of instances, which is why leave-one-repository-out is the evaluation protocol for

v_{0}

(§ C.7 ) and why repository-conditional calibration is used for conformal sets (§ C.8 ). Higher is better.

Component	Type	Estimated by	Implied action
$U_{\mathrm{task}}$	aleatoric, irreducible	residual; graph ambiguity covariates	ask the user to disambiguate
$U_{\mathrm{policy}}$	aleatoric, reducible	between-rollout variance at fixed $q$	resample and aggregate
$U_{\mathrm{model}}$	epistemic	posterior variance through the link	abstain, escalate, collect data

Appendix

Table 11: The three components of Eq. equation 16 . Each answers a different question, so conflating them discards the decision-relevant content: a system that reports only total uncertainty cannot distinguish “the request is ambiguous, ask the user” from “the model is out of its depth, escalate”.

Figure 8: The uncertainty decomposition, validated synthetically and unvalidated empirically . The panels report the forecasting behaviour of the left endpoint

v_{0}

on the real corpus (

n=499{}

, leave-one-repository-out AUROC 0.481, ECE 0.015) alongside the synthetic self-test in which the generating process is known. On the real corpus the risk–coverage curve is essentially flat at the base error rate, which is the signature of a calibrated but uninformative predictor and is why we report AURC and AUROC rather than ECE alone. On the synthetic process the decomposition behaves as specified, with epistemic uncertainty inflating

8.1{}\times

outside the training support. The contrast is the point: the machinery works where the truth is known and does not deliver a usable forecast where it is not.

Quantity	Pre-registered	Outcome
$D_{1}$ top-1 disagreement	$[40\%,75\%]$ , point $55\%$ ; falsified if $<20\%$	criterion retired, not passed : observed 97.1% (68 $/$ 70), CI [92.9%, 100%]. We initially recorded this as “direction confirmed”; that reading is withdrawn, because the null itself sits near 97.1% once the reference is known to be noise (§ C.10 )
$D_{3}$ displacement	$[+0.25,+0.40]$ ; falsified if $\leq 0$	above interval : +0.418 over $n=68{}$ , $z=6.07{}$ , $p<0.001$
Collider isolation	$[+0.15,+0.30]$ ; sign is the substantive claim	above interval : +0.537 over $n=60{}$ , $z=6.70{}$ , $p<0.001$ . Sign confirmed; magnitude implies descendant conditioning accounts for most of the displacement
Wilcoxon $p$	expected to be uncomputable at small $n$	superseded: the full attribution sweep makes it computable, and both signed-rank tests reject at $p<0.001$
Degenerate $\Delta\equiv 0$ rate	$30$ – $40\%$	better than predicted : 22.9% (16 $/$ 70). An early two-episode reading suggested $50\%$ ; the full sweep does not support it
$U_{\mathrm{policy}}$ estimable	requires $K\geq 2$ ; not available from the main sweep	met on the $K{=}4{}$ subset: $U_{\mathrm{policy}}=0.049{}$ over 23 instances

Appendix

Table 12: The pre-registration record and its outcomes. Three directional predictions held at magnitudes above the intervals we fixed in advance, which we report as the intervals having been too conservative rather than as successes. Four entries did not hold as stated: the monotone entropy decay is contradicted, the forecasting endpoint failed its usefulness threshold, the degenerate-outcome rate came in below the predicted range, and the

D_{1}

criterion was ill-posed.

Process-Based Agent Evaluation

See all 23 papers

Jun 15, 2026

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique that is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for auditing and evaluating agents for how they approach tasks at a procedural level given their traces in a top-down fashion. We believe this work has a range of applications to help developers work with and program coding agents, such as task-aware model routing, agent monitoring, and finer-grained cost analysis.

Hamidah Oderinwale

Work done with support from Taste Labs

AI Coding Agents AI Agent Evaluation Language Model Fingerprinting Agent Trajectory Analysis Process-Based Agent Evaluation

Ask PDF

Aug 14, 2026

Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model

AI coding agents are commonly evaluated as models but deployed as systems. Their reliability depends not only on model capability, but on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. This monograph examines those boundaries and develops a framework for evaluating and operating coding agents reliably. It synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis. Across this evidence, many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes. Evaluation and operation are treated as a dependency chain in which weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions. The monograph contributes a versioned catalog of 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads; an evidence ledger; a framework for dependency and repair asymmetry across the agent lifecycle; measurements and failure cases from operated agent systems; runnable evaluation and reliability protocols; and five reusable agent skills with evidence maps. Together, these provide a system-level methodology for distinguishing model capability from infrastructure effects, designing defensible evaluations, and building systems that recover safely when components fail. The review is structured rather than exhaustive, evidence strength varies by topic, and results depend on workload and configuration. The methods record which search lanes were executed, which remain unexecuted, and limits on evidence-grading claims.

Stephanie Jarmak

AI Agent Reliability Coding Agents AI Agent Evaluation Software Engineering Agents

Ask PDF

May 12, 2026

An Empirical Study of Automating Agent Evaluation

Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%.

Kang Zhou, Sangmin Woo, Haibo Ding +15

AWS AI Labs

Agent Evaluation AI Agent Benchmarks Automated Agent Evaluation Automated Evaluation

Ask PDF

Abstract

Process-Based Agent Evaluation

Explore similar work

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model

An Empirical Study of Automating Agent Evaluation