Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents

Xueping Gao

cs.CLAug 17, 2026

Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents

Authors: Xueping Gao

Organizations: Alibaba Cloud Hangzhou, China

Abstract

AI coding agents need more than relevant snippets: they need business semantics, validation evidence, relations, and assurance that their context is current. Existing systems usually infer or externalize this knowledge through retrieval, summaries, graphs, rules, or reverse specifications. We investigate a complementary representation in which selected code units directly carry agent-usable knowledge. We introduce Executable Code Knowledge (ECK) and define an Executable Code Knowledge Unit (ECKU) as a source-bound object combining stable identity, semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and a query interface. Our Python prototype supports code-local authoring, manifest export, evidence execution, exact changed-line impact, freshness checking, and agent-facing projections. Across three real Python repositories and 26 controlled patch tasks, direct ECK provides executable test coverage for 11/11 evidence-bearing tasks and exact selectors for 9/11; hiding declared evidence reduces exact recovery to 1/11 (paired exact McNemar p=0.0078). ECK-derived rules recover 11/11 exact selectors, showing that rules are effective delivery artifacts while ECK supplies source binding, validation state, impact, and freshness. Exact changed-line impact matches independently authored labels on all 26 patches (12 unit links; precision, recall, and F1 all 1.000). AST-bounded fingerprints classify 50 positive changes and 17 unrelated same-file controls correctly, whereas static rules snapshots detect none of the 50 stale cases. Model-backed patch-review and cross-layer studies measure projection fidelity rather than independent impact discovery. These results support a hybrid architecture: retrieval for coverage, ECK for source and evidence governance, and projections for delivery.

Figures & tables

Repository	Direct ECKUs	With Evidence	Controlled Patches
python-dotenv	6	6	10
python-slugify	5	4	8
tomli	6	6	8
Total	17	16	26

Table 1. Evaluated repositories and controlled patches.

Method	File Recall	Exact Selector Recall*	Executable Coverage Recall*
Plain	0.500	0.000	0.000
BM25 Chunk	0.981	0.455	0.455
EKP Retrieval	0.846	0.000	0.636
Rules Context	0.865	1.000	1.000
Direct ECK without Evidence	0.981	0.091	0.636
Direct ECK	0.981	0.818	1.000

Table 2. Validation-planning results with Qwen2.5-Coder-32B-AWQ.

Method	File Recall	Exact Selector Recall*	Executable Coverage Recall*
Plain	0.904	0.000	0.273
EKP Retrieval	0.904	0.091	0.636
Direct ECK	0.962	0.727	0.727

Table 3. Validation-planning results with the 7B model.

Metric	Value
Direct ECKUs	17
Direct ECKUs with Evidence	16
Extractive EKPs	169
Patches Touched by Direct ECK	11
Patches Touched by EKP-RAG	26
Direct Validation Plans	11

Table 4. Direct-ECK and extractive coverage across 26 patches.

Direct-impact metric	Value
True-positive unit links	12
False-positive unit links	0
False-negative unit links	0
Micro precision / recall / F1	1.000 / 1.000 / 1.000
Exact-set accuracy, all patches	1.000
Exact-set accuracy, positive patches	1.000

Table 5. Direct-impact accuracy against independently authored labels.

Method	Parsed	Strict JSON	Avg Prompt Chars	File Recall	ECK ID Recall	Exact Selector Recall	Executable Coverage Recall
Plain Patch	0.923	0.923	1344	0.923	0.000	0.000	0.273
BM25 Patch	0.962	0.962	7443	0.962	0.591	0.364	0.636
Rules Patch	0.923	0.923	4056	0.923	0.000	0.818	0.818
Direct ECK Patch	1.000	1.000	3875	1.000	1.000	1.000	1.000

Table 6. Model consumption of patch evidence reports.

Method	ECK ID Recall 95% CI	Exact Selector Recall 95% CI	Executable Coverage Recall 95% CI
BM25 Patch	0.591 [0.273, 0.864]	0.364 [0.091, 0.636]	0.636 [0.364, 0.909]
Direct ECK Patch	1.000 [1.000, 1.000]	1.000 [1.000, 1.000]	1.000 [1.000, 1.000]
Plain Patch	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]	0.273 [0.000, 0.545]
Rules Patch	0.000 [0.000, 0.000]	0.818 [0.545, 1.000]	0.818 [0.545, 1.000]

Table 7. Bootstrap intervals for ECK-ID and validation recall.

Method	Avg Prompt Chars	ECK ID Recall	API Recall	DB Field Recall	Business Rule Recall	Exact Selector Recall	Executable Coverage Recall
Plain	1591	0.000	0.559	0.108	0.059	0.118	0.118
BM25	9749	0.588	0.971	0.435	0.206	0.706	0.706
Rules	4947	0.000	0.971	0.617	0.471	1.000	1.000
Rules with IDs	5300	1.000	0.971	0.631	0.382	1.000	1.000
Direct ECK	6930	1.000	0.971	0.531	0.294	1.000	1.000
ECK Projection	4586	1.000	0.971	0.832	1.000	1.000	1.000

Table 8. Cross-layer projection fidelity.

Perturbation	Expected	Observed
source body	stale: source	stale: source
agent-facing knowledge	stale: knowledge	stale: knowledge
contract	stale: contract	stale: contract
evidence	stale: evidence	stale: evidence
unchanged	fresh	fresh
unrelated edit, same file	fresh	fresh

Table 9. Controlled freshness perturbations.

Perturbation	Cases	Correct	Accuracy
source body (positive)	17	17	1.000
agent-facing knowledge (positive)	17	17	1.000
evidence (positive)	16	16	1.000
unrelated same-file edit (negative)	17	17	1.000
total	67	67	1.000

Table 10. Real-repository freshness perturbations.

Capability	Retrieval / Code Search	Rules / Memories	Direct ECK	ECK Projection
Broad repository coverage	high	medium	selective	selective
Natural agent delivery	medium	high	medium	high
Stable code-unit identity	low	possible as generated text	high	high
Executable validation evidence	indirect	possible as text	first-class field	first-class field
Patch-to-knowledge impact	indirect	low	first-class operation	first-class operation
Source/knowledge/evidence freshness	low	low	first-class operation	first-class operation

Table 11. Capability comparison across context representations.

Purpose	Script or Artifact
ECK unit discovery, query, validation, freshness, impact, report	eck/*.py
Planning experiment	experiments/run_agent_context_experiment.py
Patch-review experiment	experiments/run_patch_review_experiment.py
Patch-review reanalysis and bootstrap CI	experiments/analyze_patch_review_results.py
Rules staleness comparison	experiments/compare_rules_staleness.py
Positive/negative freshness controls	experiments/run_eck_freshness_perturbation.py and experiments/run_eck_real_repo_freshness.py

Table 12. Artifact organization and reproduction entry points.

Explore similar work

Jul 30, 2026

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification. To expand this supply, we present Change2Task, a system grounded in repository history that converts merged pull requests into verified tasks on healthy modern revisions of the same repository. It aligns historical evidence with evolved code, reconstructs task states through Patch Reversal, Code Mapping, or Agent Reconstruction, and validates the lifecycle from a healthy base to a task state and a restored state. By deriving multiple tasks grounded in developer evidence from maintained environments, Change2Task provides executable data for coding agent training and evaluation while reducing repeated environment setup, storage, and task construction effort. We evaluate the system through five common and widely adopted coding agent task families: Bug Fix, Feature Addition, Test Generation, Application Programming Interface Migration, and Security Repair. Starting from 1,130 source changes eligible for construction, Change2Task achieves 79.6% verified task construction success across these task families. On a matched candidate set, it recovers 29.2% more verified tasks than a construction baseline based on pull requests. Historical and reconstructed cases achieve up to 98.0% matched outcome agreement under agent evaluation, while reuse of modern bases reduces measured expenditure across the complete pipeline by 10.8%.

Haomin Qi, Xingliang Wang, Xuanqi Gao +9

1Microsoft · 3Zhejiang University · 4Xi’an Jiaotong University +1

Code Generation Benchmarks Agent Benchmarks

Ask PDF

Jul 28, 2026

TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning

Contemporary LLM-based coding agents produce code as black-box outputs: the rationale behind each line is hidden, the evolution of the code through benchmark-driven repair is ephemeral, and post-hoc auditing is impossible. We present a code generation concept that addresses these shortcomings through three complementary mechanisms: (i) a relational snippet-history schema that records, per repair event, the benchmark reference, round number, failure text, and LLM explanation, enabling full provenance queries; (ii) a browser-based visualisation tool that renders this history as heat-mapped, hover-annotated source code; and (iii) a competitive fractional position-key indexing scheme with tree-node delimiters that assigns stable, lexicographically-ordered identifiers to each code snippet, enabling fine-grained tracking without disrupting surrounding lines. We evaluate TraceCoder on 30 algorithmic programming tasks spanning string processing, mathematical computation, and data-structure manipulation, across two provider configurations. Of these, 10 exhaust the 6-iteration budget on tasks with subtle edge-case behaviour. Mean Chg% reaches 30%, three in ten code snippets carry a traceable repair-event row, compared to 21% when using Gemini 2.0 Flash as sole provider on a 20-task subset. Three detailed case studies demonstrate how the system explains which specific benchmark failures shaped each line of the final program. The proposed mechanism makes the internal "narrative" of automated code generation auditable and replayable, a property essential for trust and accountability in production deployments.

Rwaida Alssadi, Muntaser Syed, Balaji Kasula +6

Florida Institute of Technology

Data Provenance LLM Auditing Code Generation

Ask PDF

May 19, 2026

Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study

As autonomous coding agents see rapid adoption, their evaluation has primarily focused on task completion rates holding the target codebase fixed. This leaves a critical question unanswered: does the structural and stylistic quality, or ``cleanliness'' of the underlying code affect an agent's ability to navigate and modify it? To isolate the effect of code cleanliness from agent capability, we introduce an evaluation protocol built around minimal pairs: repositories that match on architecture, dependencies, and external behaviour, but differ on static-analysis rule violations and cognitive complexity. The pairs are constructed in both directions, by agent pipelines that either degrade a clean repository or clean a messy one. We author 33 tasks across six such pairs, evaluated through hidden tests at the application's public surface. Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development, shaping the computational cost and navigational efficiency of coding agents. Code cleanliness joins model choice, harness, and prompting as a factor that materially affects agent behaviours.

Priyansh Trivedi, Olivier Schmitt

SonarSource

Coding Agents AI Agent Evaluation AI Agent Benchmarks

Ask PDF