What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports

Guoyang Xu; Rui Zhang; Xiuyu Wu; Hao Chen

cs.AIOct 14, 2026

What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports

Authors: Guoyang Xu, Rui Zhang, Xiuyu Wu, Hao Chen

Organizations: Tencent

Abstract

Human supervisors rely on coding agents' reports to assess progress, verify repairs, and decide when to intervene. But a supervisor cannot investigate a failure they were never told occurred. Failed attempts, workarounds, and unresolved risks may disappear from routine reports, leaving supervisors unaware of issues that warrant scrutiny. We conduct CTXGAP, a measurement study quantifying the gap in task-state recoverability between agents' accumulated user-facing reports and their full recorded execution contexts. Across 37 coding tasks and 4,799 valid probes, agents answer 93.6% of task-state questions on demand; full-context readers recover 92.7%, while report-only readers recover 52.0%, a 40.7-point gap. At the final checkpoints of 30 verified fixes, the gap remains 34.8 points. Human annotation identifies states mentioned without enough supporting detail as the largest category of unrecovered information, and model-assessed supervision-relevant states retain substantial gaps. Full terminal scrollback improves recovery to 73.6% but still trails full context. Under matched report lengths, context-informed rewrites improve recovery by about eight points, without a detectable targeted-over-natural advantage. Positive gaps persist across harnesses and independent DeepSeek execution and measurement. CTXGAP establishes report-based task-state recoverability as a complementary evaluation dimension.

Figures & tables

Figure 1: CTXGAP overview. Execution-grounded task state can be unrecoverable from accumulated reports; paired probes quantify the 40.7-point main-sample gap between $T_{\mathrm{full}}$ and $U$ (37 tasks; 4,799 valid probes). Panel 4 illustrates two archived cases—an unreported Django compatibility risk and an earlier pytest verification failure—with original probe judgments $U$ : 0/3 and $T_{\mathrm{full}}$ : 3/3 (Appendix A.3 ). Panels 1–3 illustrate the paired-view measurement; Panel 5 shows the evaluated application settings.

View / setting	Accuracy	$\mathrm{Acc}(T)$	Gap	95% CI
Main measurement (37 tasks; 4,799 probes)
Reports $U$	.520	.927	.407	[.373, .438]
Scrollback $V$	.736	.927	.191	—
Actor fork (aux.)	$\mathrm{Acc}(A)=\mathbf{.936}$ [ $.929,.944$ ]
Fixed at $k=4$ †	.571	.919	.348	[.282, .413]
Cross-harness check (12 shared issues)

Table 1: Execution–report task-state recoverability.

U

: accumulated reports;

V

: full terminal scrollback;

T

: full recorded execution context. Gap is

\mathrm{Acc}(T)-\mathrm{Acc}(\mathrm{view})

; actor self-answering is auxiliary. † Final checkpoints of 30 hidden-test-fixed tasks (1,172 probes). Main estimates use category-weighted probes and issue-cluster bootstrap CIs; cross-harness rows use task-equal means, distinct trajectories/probes, and per-issue

t

intervals. The scrollback gap is a point estimate; reading-view intervals are in Appendix B .

Component	Validity evidence	Study scope
Probe eligibility	4,799 of 4,855 retained; filter changes $\mathrm{Gap}_{\mathrm{view}}$ by +0.56 pp	Eligibility determined after full-context scoring
Answer scoring	Alternative rubric: +1.1 pp [ $-1.3$ , +3.6]; dual-rater machine–human difference: +1.39 pp [ $-1.37$ , +4.54]	Rubric and dual-rater audits on separate samples
Conveyance validity	195 stratified atoms: mention $\kappa=.632$ , sufficiency $\kappa=.638$	Mentioned but insufficient: 55.4% / 59.2% of weighted gap mass

Table 2: Validity evidence for probe eligibility, answer scoring, and report conveyance. Brackets indicate 95% task-cluster CIs; the dual-rater contrast uses task-equal aggregation. Full protocols appear in Appendices G and A.10 .

Figure 2: Recovery by reading depth across 11 views (37 tasks, 4,799 probes). Error bars are 95% task-cluster bootstrap CIs. Purple annotations indicate gaps relative to actor self-answer accuracy (0.936); views need not be nested, and the two rightmost views contain internal execution context. Detailed estimates appear in Table 14 .

Figure 3: RQ1 information-transfer diagnosis. (a) Stratified inverse-probability-weighted gap-mass estimates from two independent annotators (195 atoms), with task-cluster bootstrap 95% CIs. (b) Unaligned atom counts grow across checkpoints while normalized

\mathrm{Gap}_{\mathrm{actor}}

is non-monotonic. Automatic semantic classes are detailed in Appendix B.3 .

Report	NI recovery	$n_{NI}$	Overall	$n_{all}$	Chars	$n_{cost}$
Reports only	0.017	54	0.512	120	8,480.5	120
Natural	0.031	54	0.518	120	1,573	120
Structured	0.281	54	0.597	120	3,184	120
Targeted (matched)	0.110	50	0.564	112	1,460.5	112
Targeted	0.602	50	0.686	112	5,693.5	112

Table 3: Held-out reporting results. Recovery is the mean over available checkpoint-level scores. Non-inferable (NI) and overall outcomes have different denominators; cost is the median number of report characters.

Appendix figures & tables66 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Tasks	Units
Main sample	37	148 checkpoints
State probes	37	4,799 probes
Supervision contrast	72	284 probe-linked units
Reporting development	46	184 checkpoints
Reporting held-out	30	120 checkpoints
Cross-harness	12	36 trajectories

Appendix

Table 4: Tasks are unique issue IDs; units differ by experiment. The 76-task inventory selects one run per issue. Cross-harness has three trajectories per issue; cross-executor has two (reusing the DSH–HY4 runs). Splits and clusters use issue IDs.

Measurement	Estimate	95% CI
Actor $A$	0.936	[0.929, 0.944]
Full context $T$	0.927	[0.913, 0.938]
Reports $U$	0.520	[0.491, 0.551]
$\mathrm{Gap}_{\mathrm{actor}}$	0.416	[0.386, 0.444]
$\mathrm{Gap}_{\mathrm{view}}$	0.407	[0.373, 0.438]

Appendix

Table 5: Main-sample weighted accuracies and gaps. Main

\mathrm{Gap}_{\mathrm{view}}

interval uses 10,000 issue-clustered percentile bootstrap draws of the category-weighted pooled estimator (seed 0); other cells retain their recorded interval protocols.

Checkpoint	Instruction
1	Investigate; do not modify code
2	Implement a fix
3	Run tests and report the result
4	Assess remaining risk

Appendix

Table 6: Common instruction sequence and protocol parameters.

Parameter	Value
Minimum atom importance	2
Full / partial / no credit	1 / 0.5 / 0
Probe answer samples per view	3
Measurement thinking mode	Disabled
Headline view-gap bootstrap resamples	10,000
Confidence level	95%

Appendix

Table 7: Fixed measurement parameters and reported collection checks.

Checked stage	Runs checked	Distinct windows
Atom extraction	32	1
Probe generation	32	1
Judge	32	1
Observer: reports ( $U$ )	32	1
Observer: execution ( $T$ )	32	1
Observer: task prior	32	1

Appendix

Table 8: Per-run logged-request conformance within the mixed-version p2 batch (18 v3.0, 14 v3.2). The common hash in each row applies only to the extracted static-template window, not to the entire request.

Subset	Valid probes	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T_{\mathrm{full}})$	$\mathrm{Gap}_{\mathrm{view}}$
Frozen 37	4,799	.5205	.9271	.4066
Additional 39	5,387	.4748	.9372	.4624
Expanded 76	10,186	.4950	.9328	.4378

Appendix

Table 9: Cohort sensitivity under the same category-weighted micro estimator. The 76-task row is an expanded observational population, not a replacement for the frozen 37-task headline.

Subset	Valid $n$	$\mathrm{Acc}(U)$	$\mathrm{Gap}_{\mathrm{view}}$ [95% CI]
Fixed; all $k$	3,914	.5129	.4118 [.3716,.4478]
Fixed; $k=4$	1,172	.5705	.3479 [.2816,.4126]
All runs; $k=4$	1,450	–	.3451 [.2902,.3979]

Appendix

Table 10: View-gap recovery by outcome and checkpoint, using the primary category-weighted probe estimator and issue-clustered bootstrap. Final-checkpoint fixed tasks have

\mathrm{Acc}(T_{\mathrm{full}})=.9185

; their gap is positive for 28 of 30 issues. The 30 eventually fixed tasks have

\mathrm{Acc}(T_{\mathrm{full}})=.9246

across all checkpoints.

Subset	Valid $n$	Tasks	$\mathrm{Gap}_{\mathrm{view}}$ [95% CI]
Primary cohort	4,799	37	.4066 [.3733,.4381]
Drop affected run	4,653	36	.4141 [.3824,.4439]
Drop 86 mismatches	4,713	37	.4110 [.3796,.4404]

Appendix

Table 11: Sensitivity to the probe–atom binding inconsistency. All estimates use the original frozen scores, original probe-category weights, and the same issue-clustered protocol; no probe IDs or references are rewritten.

Recorded stage	Calls	Explicit temperature
Atom extraction	226	none (226 unknown)
Probe + reference generation	520	0.0 (211); 309 unknown
Observer answers	8,258	0.7 (697); 7,561 unknown
Judge / coverage / audits	9,929	0.0 (1,064); 8,865 unknown

Appendix

Table 12: Archived 37-run model-call audit, including associated coverage and rescoring work. All stages use HY4-preview; the reference is emitted in the probe-generation call. No top_p or top_k values are logged. “Unknown” means omitted from the recorded request, not zero or a verified server default. Counts are logged calls, not independent judged probes.

View (weighted micro)	Accuracy	Share [95% CI]
Reports $U$	.5205	–
Terminal scrollback $V$	.7359	53.0% [47.6, 57.8]
Execution, no reasoning $T_{\mathrm{minus}R}$	.8902	37.9% [33.1, 43.7]
Full execution $T_{\mathrm{full}}$	.9271	9.1% [6.0, 12.1]
Full-minus-report gap	.4066	100%

Appendix

Table 13: Exact telescoping decomposition of the weighted main-sample report-to-full-context gap (

n=4{,}799

probes, 37 tasks). Each view row’s share is the accuracy increase from the preceding row divided by .406582; step increases are .215436, .154222 and .036924, respectively. Share intervals use paired task-cluster bootstrap (2,000 resamples). The displayed view accuracies are weighted micro means. A share is not the fraction of facts demonstrably contained only in that view.

View	Mean chars	Accuracy	95% CI	$\mathrm{Gap}_{\mathrm{actor}}$
Latest report	2,565.1	0.239	[0.219, 0.263]	0.697
All turn reports	6,564.4	0.520	[0.491, 0.551]	0.416
+ Narration	7,532.8	0.545	[0.520, 0.574]	0.391
+ Exec. headers	13,064.8	0.639	[0.610, 0.669]	0.297
+ Last 3 outputs	14,553.5	0.657	[0.635, 0.683]	0.279
+ Last 10 outputs	18,256.3	0.690	[0.666, 0.715]	0.247

Appendix

Table 14: Reading conditions and summary additions. The two recent-tool-call conditions branch from reports; full context extends beyond terminal visibility. Context cost is the mean number of characters.

$k$	Fresh	Earlier	Unaligned	$\mathrm{Gap}_{\mathrm{actor}}$	95% CI
1	9.423	0.014	9.436	0.426	[0.375, 0.480]
2	11.546	7.480	19.027	0.472	[0.439, 0.505]
3	3.057	17.984	21.041	0.436	[0.390, 0.482]
4	3.411	18.869	22.280	0.359	[0.317, 0.403]

Appendix

Table 15: Unaligned state and actor gap by checkpoint. Unaligned counts are fresh plus earlier-turn atoms; the accuracy gap is a separate normalized measurement.

Component	Share	Atoms
Not conveyed	0.711	1397
Reported, not recovered	0.151	294
Wrong report answer ∗	0.108	213
Latest-only partial †	0.028	65
Stale	0.002	5
Total	1.000	1974

Appendix

Table 16: Automatic semantic decomposition of main-sample gap mass. “Not conveyed” is the classifier’s non-conveyance class, not literal non-mention; human mention and sufficiency are measured separately (Appendix B.4 ). Counts are gap atoms; shares are weighted. ∗ Wrong-answer labels take precedence over non-conveyance. † Same-checkpoint partial recovery under the latest-report view.

Source class	Semantic	Provenance-flag split
Not conveyed	0.711	0.867
Reported, not recovered	0.151	0.020
Wrong report answer	0.108	0.108
Latest-only partial	0.028	0.002
Stale	0.002	0.002

Appendix

Table 17: Semantic-class judgments and extraction-provenance flags. The provenance flag indicates extraction from assistant_text , a distinct measurement from report conveyance.

Quantity	Share
Old gap mass	0.620
Old, not traced to assistant_text	0.979
Old, judged not conveyed	0.942

Appendix

Table 18: Older-turn atom diagnostics. The source-provenance flag does not measure literal mention in reports; only the frozen semantic judgment refers to report conveyance.

Category	Atoms	Total share	Not conveyed	Reported, unrecovered	Wrong report answer	Latest-only partial	Stale
Facts	632	0.428	0.687	0.187	0.098	0.028	0.000
Failures	373	0.166	0.894	0.028	0.074	0.004	0.000
Progress	332	0.159	0.545	0.221	0.169	0.052	0.012
Environment	205	0.082	0.825	0.068	0.090	0.014	0.004
Risks	128	0.050	0.666	0.157	0.149	0.027	0.000
Decisions	99	0.037	0.678	0.177	0.094	0.051	0.000

Appendix

Table 19: Complete semantic category decomposition using the main-sample population. "Not conveyed" denotes automatic semantic non-conveyance. All values except counts are proportions.

Figure 4: Supplementary RQ1 category decomposition. (c) Within-category gap composition and (d) each category’s contribution to total semantic gap mass. This complements the omission and accumulation evidence in Figure 3 .

Type	Frozen share ( $n$ )	Omission first ( $n$ )
Not conveyed	.7112 (1397)	.7984 (1568)
Wrong report answer	.1082 (213)	.0211 (42)
Reported, not recovered	.1506 (294)	.1506 (294)
Latest-only partial	.0277 (65)	.0277 (65)
Stale	.0022 (5)	.0022 (5)

Appendix

Table 20: Classification-precedence sensitivity on the same 1,974 semantic gap atoms, keeping all original weights and scores. Only 171 atoms move from misled to omitted ; weighted mass is conserved and each column sums to one at full precision (displayed shares are rounded).

Operational rule	“Decay” $n$	Mass (%)	Residual $n$
Frozen latest-only	65	2.77	294
Zero included	307	16.04	52
Prior recovery lost	121	6.29	238 †

Appendix

Table 21: Sensitivity of the conveyed, non-misled, non-stale pool (359 atoms) under alternative decay rules. The residual includes reported-but-unrecovered atoms; † also includes 163 atoms without a comparable earlier checkpoint. All rules keep omission fixed at 1,397 atoms (71.12% of total weighted gap mass). The longitudinal row is a recovery-loss diagnostic, not an adjudicated forgetting rate.

	Annotator A		Annotator B
State for report-only probe	Mass share	95% CI	Mass share	95% CI
Not mentioned	.4205	[.314,.535]	.3371	[.247,.426]
Mentioned, insufficient	.5543	[.439,.658]	.5917	[.493,.691]
Sufficient, not recovered	.0252	[.011,.045]	.0371	[.019,.061]
Uncertain	.0000	—	.0341	[.004,.073]

Appendix

Table 22: Independent human measurement of report mention and probe-conditioned evidence sufficiency in 195 stratified gap atoms. Mass shares are inverse-stratum-probability weighted estimates relative to the frozen 1,974-atom gap population; 95% intervals use task-cluster bootstrap. Annotators are shown separately; both confidence intervals for “mentioned, insufficient” include 50%. Values quantify where gap mass is associated with information states , not causal contributions or accuracy conditional on sufficiency.

	A		B
Frozen omitted	Share	CI	Share	CI
Not mentioned	.5254	[.385,.669]	.3927	[.273,.517]
Mentioned, insufficient	.4746	[.332,.616]	.5748	[.445,.700]
Sufficient	.0000	—	.0000	—

Appendix

Table 23: Human distinction within the frozen omitted stratum (70 sampled atoms). Uncertain residual is not displayed; two separately weighted raters are not a consensus.

Group	$n$	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T)$	Gap [95% CI]
HY4 relevant	151	.553	.906	.353 [.266,.435]
HY4 irrelevant	42	.368	.900	.532 [.421,.627]
DS relevant	135	.546	.910	.364 [.259,.463]
DS irrelevant	49	.377	.873	.496 [.379,.606]
Both relevant	125	.553	.914	.361 [.260,.462]
All scored	195	.508	.901	.393 [.316,.467]

Appendix

Table 24: Descriptive recoverability in the stratified audit sample. Labels are AI-assessed decision relevance, not human ground truth. CI calculations cluster by task.

Observer	$O$	$M$	$E$	$E-M$ [95% CI]	$M-O$ [95% CI]
HY4-preview	.303	.258	.535	+.278 [.205,.347]	-.045 [-.111,.020]
DeepSeek	.323	.343	.525	+.182 [.119,.245]	+.020 [-.016,.056]
GLM	.475	.495	.601	+.106 [.037,.177]	+.020 [-.043,.077]

Appendix

Table 25: Targeted intervention on all 99 selected candidates (33 tasks); full-pool estimates with task-clustered intervals. All figures are conditional on the human-screened pool, not estimates of overall gap composition.

Quartile	Mean overlap	$E-M$	95% CI
Q1	.005	+.028	[-.036,.086]
Q2	.053	+.113	[.018,.213]
Q3	.208	+.167	[.080,.257]
Q4	.479	+.440	[.300,.594]

Appendix

Table 26: Exploratory evidence–reference lexical-overlap gradient in the 99-candidate intervention pool. Quartiles should not be interpreted as randomized evidence-quality groups.

Metric	Frozen	Adjusted	$\Delta$ [95% CI]
$\mathrm{Gap}_{\mathrm{actor}}$	.4159	.4244	+.0085 [.0018, .0154]
$\mathrm{Gap}_{\mathrm{view}}$	.4066	.4153	+.0088 [.0021, .0153]

Appendix

Table 27: Importance post-stratification in the 37-run main sample. Intervals use a run-clustered bootstrap. Frozen values are reproduced by the category-only estimator before changing weights.

Harness	Orig.	Adjusted	CI ( $\Delta$ )
Claude Code	.4070	.4216	[.0053, .0260]
Pi	.3903	.4026	[-.0003, .0238]
DSH	.4898	.5144	[.0130, .0362]

Appendix

Table 28: View-gap sensitivity for each harness in the separate 12-task sample. Intervals use task-clustered bootstrap of the within-harness change; no cross-harness ranking is inferred.

Audit	Items	Annotators
Conveyance wave 1	20	3
Probe-sufficiency machine audit	403	—
Stratified human sufficiency	45	2
Human-consensus subset	32	2
Question-aligned re-scoring	240	—
Disputed items retained as errata	14	—

Appendix

Table 29: Audit sample sizes and recorded defects from the study.

Audit label	Class share	Gap share	Items
Partial support	0.797	0.120	235
Full support	0.173	0.026	50
No support	0.024	0.004	8
Contradictory	0.005	0.001	1
Total	0.999	0.151	294

Appendix

Table 30: Probe sufficiency within the reported-but-unrecovered class. The original probe audit covers a broader set of 403 probes; this table refers to its 294-item class subset. Class shares and weighted mass are reproduced at the precision reported by the study.

Comparison	Agreement	$\kappa$
Human–human	0.711	0.441
Machine–annotator 1	0.822	0.554
Machine–annotator 2	0.711	0.447
Machine–human consensus	0.875	0.721

Appendix

Table 31: Human calibration of probe sufficiency. Agreement and kappa are distinct measures.

Annotator / subset	Upgrades	Partial items	Rate
Annotator 1	5	35	0.143
Annotator 2	11	35	0.314
Consensus	3	24	0.125
Mean annotator rate	—	—	0.229

Appendix

Table 32: Partial-to-full reclassification in the calibration sample. Consensus uses the subset on which both humans agree.

Diagnostic	Value
Wave-1 Fleiss kappa	0.565
Annotator 1: stricter / looser	5 / 2
Annotator 1: exact p	0.453125
Annotator 2: stricter / looser	11 / 0
Annotator 2: exact p	0.000977

Appendix

Table 33: Additional annotation diagnostics. Exact tests compare machine-stricter and machine-looser discordances.

Scenario	Upgrade rate	Gap share
Machine	0.000	0.034
Consensus adjustment	0.125	0.054
Mean-annotator adjustment	0.229	0.070

Appendix

Table 34: Sensitivity of the fully supported residue over the complete audited population. This population differs from the class subset in Table 30 . These are reclassification scenarios.

Figure 5: Observer-side residue under machine, consensus-adjusted, and mean-annotator-adjusted judgments, using the complete audited population.

Quantity	Original	Re-scored	Change
$\mathrm{Acc}(U)$	0.514	0.540	+0.026
$\mathrm{Acc}(A)$	0.935	0.972	+0.037
Gap	0.421	0.432	+0.011
Gap $\Delta$ 95% CI	—	—	[-0.013, +0.036]

Appendix

Table 35: Question-aligned re-scoring on the original study stratified audit sample. The gap-change interval is reported for the paired difference.

Human rule	Task-paired $\Delta$	95% CI
Two-rater mean (partial=0.5)	+0.0139	$[-0.0137,+0.0454]$

Appendix

Table 36: Machine-minus-human view-gap contrast under a common partial-credit rubric. The frozen sample contains 360 non-forced probes from 37 tasks. The CI uses 2,000 task-clustered bootstrap samples (seed 0).

Figure 6: Simulated supervision: simulated supervisory instruction changes under report-only (

U

), omitted-fact reveal (

G

), and matched-known reminder (

N

) conditions. The paired contrast is

G-N=+18.9

percentage points (task-clustered 95% CI [+10.2, +28.0]). This is an instruction-change endpoint, not a measured improvement in final task success.

Measurement	Estimate	95% CI
Restored omission $r_{G}$	0.718	—
Matched-known $r_{N}$	0.525	—
Paired $G-N$	0.189	[0.102, 0.280]

Appendix

Table 37: Instruction changes relative to the report-only arm. Arm rates are unit-level; the paired contrast uses task means.

Quantity	Count	Base	Rate
Non-inferable omissions	241	286	0.843
Instructions changed	206	286	0.720
Uptake flag	149	286	0.521
Changed + uptake	138	286	0.483
Fact-targeting check	1	7	0.143
Behavior: informed wins	4	14	—

Appendix

Table 38: Supervision diagnostics. The full inferability sample and the matched-control sample are separate. Uptake flags and their intersection with instruction change are reported separately.

Report condition	Accuracy
$A$ : original accumulated $U$	.6716
$B$ : natural rewrite	.7545
$C$ : targeted rewrite	.7566

Appendix

Table 39: E2 equal-total-length replay: 129 paired units, 37 runs, 4,175 valid probes. Task-cluster intervals.

C-B

is not significant;

A

is the original agent report, whereas

B

and

C

access full execution context. The E2 subset differs from the main-sample measurement population.

Parameter	Value
Development gap atoms	2,116
Target category coverage threshold	0.70
Selected category coverage	0.764
Matched generation budget (tokens)	1,032
Observed matched median tokens	951
Development / held-out checkpoints	184 / 120

Appendix

Table 40: Development selection and generation budgets recorded in the study.

Figure 7: Report cost and recovery on held-out checkpoints. On this held-out added-summary comparison, targeted reports have higher recovery than natural summaries, including under a matched generation budget; this is not the equal-total-length E2 contrast. Points use available-unit recovery means and median report lengths; endpoint and cost sample sizes are reported separately in Table 3 .

Contrast	Units	Tasks	$\Delta$	95% CI
TM $-$ Nat	50	25	0.077	[0.030, 0.133]
Targeted $-$ Nat	50	25	0.568	[0.455, 0.672]
Struct $-$ Nat	54	27	0.250	[0.164, 0.347]
TM $-$ Struct	50	25	-0.188	[-0.292, -0.097]
Targeted $-$ TM	50	25	0.492	[0.378, 0.597]
Targeted $-$ Struct	50	25	0.303	[0.167, 0.420]

Appendix

Table 41: Task-paired non-inferable recovery contrasts. TM denotes targeted reporting under the natural-summary budget; Nat denotes natural summaries, and Struct denotes structured summaries.

Contrast	Units	Tasks	Delta	95% CI
matched – nat	112	28	0.049	[0.036, 0.062]
targeted – nat	112	28	0.171	[0.149, 0.194]

Appendix

Table 42: Task-paired overall recovery contrasts.

Category	Probes	Before	After
Failures	437	0.823	0.810
Rejected options	120	0.643	0.640
Environment	326	0.617	0.576
Progress	987	0.390	0.369
Assumptions	102	0.384	0.394
Facts	1866	0.376	0.377

Appendix

Table 43: Within-category actor gap before and after adding a natural summary.

Figure 8: Within-category actor gap before and after adding a natural summary to reports. All categories are shown; the horizontal axis measures a within-category accuracy difference.

Tolerance	Units	Tasks	Natural	Targeted	$\Delta$ [95% CI]
$\leq 10$ %	47	25	.6728	.6967	+.0239 [.0105,.0357]
$\leq 15$ %	79	30	.6811	.7063	+.0252 [.0145,.0349]

Appendix

Table 44: Overall recoverability under matched realized added-summary character budgets, preserving original accumulated reports

U

. Intervals use task-cluster bootstrap with three paired samples. This outcome is not the non-inferable-state subset in Table 3 .

Probe	Observer	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T_{\mathrm{full}})$	$\mathrm{Gap}_{\mathrm{view}}$
HY4	HY4	.5206	.8995	.3789
HY4	DeepSeek	.5012	.8462	.3450
DeepSeek	HY4	.4879	.8874	.3995
DeepSeek	DeepSeek	.4564	.8547	.3983

Appendix

Table 45: Cross-generator/observer comparison under one answering and judging protocol, on the same 413 fully paired atoms (37 tasks; GLM judge). All entries are unweighted atom-level means; the original main-result

\mathrm{Gap}_{\mathrm{view}}=.4066

uses a different population and scoring protocol.

Within-atom contrast (second $-$ first)	$\Delta\mathrm{Gap}_{\mathrm{view}}$	95% CI
DS vs. HY4 probe; HY4 observer	+.0206	[ $-.0216$ ,.0644]
DS vs. HY4 probe; DS observer	+.0533	[.0049,.1076]
DS vs. HY4 observer; HY4 probe	$-.0339$	[ $-.0744$ ,.0048]
DS vs. HY4 observer; DS probe	$-.0012$	[ $-.0352$ ,.0351]

Appendix

Table 46: Cross-generator/observer contrasts on 413 complete atom-matched instances. Percentile 95% CIs use 2,000 bootstrap resamples clustered by task (37 tasks; fixed seed). The second contrast excludes zero nominally , without a multiple-comparison adjustment.

Aligned-pair contrast	$\Delta\mathrm{Gap}_{\mathrm{view}}$	95% CI
DS vs. HY4 probe; HY4 observer	+.0174	[ $-.0276$ ,.0615]
DS vs. HY4 probe; DS observer	+.0348	[ $-.0151$ ,.0892]
DS vs. HY4 observer; HY4 probe	$-.0321$	[ $-.0766$ ,.0090]
DS vs. HY4 observer; DS probe	$-.0147$	[ $-.0517$ ,.0247]

Appendix

Table 47: Same-state matched subset (

n=374

, 37 tasks), with the same paired, task-cluster bootstrap estimator.

LLM label	Probe/task	$\mathrm{Gap}_{\mathrm{view}}$	95% CI
Literal-dependent	85/19	.4785	[.4011,.5509]
Non-literal	74/19	.3690	[.2902,.4436]

Appendix

Table 48: Recovery by two-model-concordant literal-detail label, under frozen probe weights and 10,000 task-cluster bootstrap draws (seed 0). The two groups jointly cover 21 tasks and overlap on 17 tasks.

Pipeline (tasks)	$\mathrm{Gap}_{\mathrm{view}}$	95% task CI
HY4 measurement (37)	.4066	[.373,.438]
DS measurement (37)	.3985	[.348,.456]
HY4 on DS execution (12)	.5468	[.462,.643]
DS on DS execution (12)	.4885	[.431,.549]

Appendix

Table 49: Independent measurement-chain checks. The 37-task rows share frozen trajectories; the 12-task rows share independently collected DeepSeek execution trajectories. Each chain constructs different atoms and probes. The 12-task intervals are sensitive to individual tasks.

	DeepSeek independent measurement				HY4 frozen measurement
Checkpoint	Valid $n$	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T_{\mathrm{full}})$	$\mathrm{Gap}_{\mathrm{view}}$	Valid $n$	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T_{\mathrm{full}})$	$\mathrm{Gap}_{\mathrm{view}}$
$k=1$	829	.557	.933	.376	789	.496	.917	.421
$k=2$	1,215	.506	.937	.431	1,223	.468	.922	.453
$k=3$	1,340	.507	.932	.425	1,337	.501	.940	.439
$k=4$	1,416	.574	.938	.364	1,450	.579	.924	.345

Appendix

Table 50: Per-checkpoint independent measurement-chain results on the same 37 frozen task trajectories (weighted valid-probe estimates). Both pipelines independently sample probes, so the rows are not matched question sets.

Subset (DS $-$ HY4)	Pooled $\Delta$	Task-equal $\Delta$
37 tasks	$-.0081$ [ $-.058,+.047$ ]	$-.0294$ [ $-.064,+.007$ ]
12 DS-executor tasks	$-.058$ [ $-.149,+.029$ ]	$-.029$ [ $-.096,+.031$ ]

Appendix

Table 51: Distinct estimands for the DeepSeek-minus-HY4 measurement-chain contrast. Intervals appear below point estimates. Pooled

\Delta

subtracts chain-weighted aggregates; task-equal

\Delta

averages within-task differences (positive in 12/37 and 5/12 tasks). CIs use task-cluster resampling; probe sets differ.

Harness	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T_{\mathrm{full}})$	$\mathrm{Gap}_{\mathrm{view}}$	95% CI
Claude Code	.525	.932	.407	[.336, .478]
Pi	.550	.940	.390	[.289, .491]
DSH	.429	.919	.490	[.417, .563]

Appendix

Table 52: Execution–report recoverability for 12 unique issue IDs and 36 harness-specific trajectories (95% CIs: per-issue

t

df=11

). The executor is HY4-preview throughout; each harness produces distinct trajectories and probes.

Figure 9: Execution–report view gaps across three coding-agent harnesses on 12 shared tasks, using one trajectory per harness and task. Error bars show task-level uncertainty; the instrument compares different trajectory-specific state probes.

Pair	Difference	95% CI
ClaudeCode – Pi	0.017	[-0.122, 0.155]
ClaudeCode – DSH	-0.083	[-0.172, 0.006]
Pi – DSH	-0.100	[-0.207, 0.008]

Appendix

Table 53: Task-paired view-gap differences. Two-sided Student

t

intervals, df = 11, without multiplicity adjustment.

Executing model	$\mathrm{Acc}(U)$	$\mathrm{Acc}(T_{\mathrm{full}})$	$\mathrm{Gap}_{\mathrm{view}}$
HY4-preview ( $s=0$ )	.4264	.9487	.5223
DeepSeek-v4-flash ( $s=0$ )	.4212	.9482	.5270

Appendix

Table 54: Cross-executor recoverability under fixed DSH, 12 tasks per model, single answer sample and task-equal aggregation. Task-bootstrap 95% CIs are [.4278, .6130] (HY4) and [.4596, .6085] (DeepSeek); paired

\Delta_{\mathrm{DS-HY4}}=+.0047

[

-.1089

+.1535

Quantity	Claude Code	Pi	DSH
View gap	0.407	0.390	0.490
Not conveyed	0.251	0.194	0.284
Reported, unrecovered	0.075	0.074	0.105
Wrong answer	0.030	0.053	0.049
Partial recovery	0.060	0.075	0.058
Unclassified	0.001	0.000	0.000

Appendix

Table 55: Cross-agent decomposition and diagnostics. Contributions are accuracy proportions; non-conveyance share is the mean per-task share of positive typed mass.

Pair	Difference	95% CI
ClaudeCode – Pi	0.057	[-0.073, 0.186]
ClaudeCode – DSH	-0.033	[-0.121, 0.055]
Pi – DSH	-0.089	[-0.198, 0.019]

Appendix

Table 56: Task-paired omission contribution differences. Two-sided Student

t

intervals, df = 11, without multiplicity adjustment.

Pair	Difference	95% CI
ClaudeCode – Pi	0.104	[-0.011, 0.218]
ClaudeCode – DSH	-0.036	[-0.146, 0.074]
Pi – DSH	-0.140	[-0.239, -0.040]

Appendix

Table 57: Task-paired omission prevalence differences. Two-sided Student

t

intervals, df = 11, without multiplicity adjustment.

Pair	Difference	95% CI
ClaudeCode – Pi	-0.001	[-0.087, 0.085]
ClaudeCode – DSH	-0.067	[-0.136, 0.001]
Pi – DSH	-0.066	[-0.161, 0.029]

Appendix

Table 58: Task-paired conditional recovery-failure differences. Two-sided Student

t

intervals, df = 11, without multiplicity adjustment.

Quantity	Count
Cells with available usage	34
Cells with partial usage	2
Cells with unavailable usage	0
Partial cell: usable / total logs	628 / 631
Partial cell: usable / total logs	530 / 531
Collection unique requests	2,545

Appendix

Table 59: Usage coverage and request counts recorded in the study.

Actor	Eff. input	Eff. output	Cache read
Claude Code	1,554,998	846,179	40,115,328
DSH	1,520,598	649,127	40,074,112
Pi	3,482,598	1,867,266	34,649,536
Total	6,558,194	3,362,572	114,838,976

Appendix

Table 60: Collection traffic by coding agent. Effective usage keeps the final successful request; actual usage includes retries with reported tokens. Cache reads are separate.

Stage	Input	Output	Cache read
Judge	22,134,005	925,035	3,107,264
Probes	21,678,906	814,186	941,440
Observer	17,420,919	5,541,630	32,200,704
Atoms	5,524,555	588,513	536,576
Semantic conveyance	2,266,075	492,617	168,000
Coverage	1,294,524	167,853	1,078,784

Appendix

Table 61: Effective measurement usage by pipeline stage.

Purpose	Input	Output
Core	53,491,784	3,674,329
Sensitivity	16,827,200	4,855,505
Other	2,813,018	150,674
Supervision	310,177	13,422
Audit overhead	218,783	1,021,353

Appendix

Table 62: Effective measurement usage by purpose. Core includes atoms, probes, report/actor observers, judging, and semantic conveyance.

Metric	Collection	Measurement	Total
Input	6,558,194	73,660,962	80,219,156
Output	3,362,572	9,715,283	13,077,855
Input+output	9,920,766	83,376,245	93,297,011
Cache read	114,838,976	41,607,360	156,446,336

Appendix

Table 63: Effective observable token totals. Cache reads are a separate token class, not added to the input+output headline.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 139 papers

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
Feb 12, 2026Thibaud Gloaguen, Niels Mündler-Sasahara, Mark Niklas Müller +2AI Coding Agents Coding Agents
Department of Computer Science ETH Zurich
Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks
Jun 1, 2026Dipesh KC, Anjila BudathokiCoding Agents Software Engineering Benchmarks
Independent Researcher · Georgia State University
Knowing When to Stop: The Evidence-Action Gap in Coding Agents
Oct 22, 2026Tianyu Chen, Xinran Zhao, Vijay Viswanathan +3AI Coding Agents Process-Based Agent Evaluation
University of Texas at Austin · Carnegie Mellon University · The University of North Carolina at Chapel Hill

Feb 12, 2026J/K move · Enter open · S save

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Thibaud Gloaguen, Niels Mündler-Sasahara, Mark Niklas Müller, Veselin Raychev+1

Department of Computer Science ETH Zurich

AI Coding Agents Coding Agents LLM Agent Evaluation LLM Context Management

Open paper AskPDF arXiv

A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files, and a novel collection of issues from repositories containing developer-committed context files. Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files. Specifically, we find that while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful. We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.

Abstract

Coding Agents

Explore similar work

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks

Knowing When to Stop: The Evidence-Action Gap in Coding Agents