Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents
Organizations: Alibaba Cloud Hangzhou, China
Abstract
AI coding agents need more than relevant snippets: they need business semantics, validation evidence, relations, and assurance that their context is current. Existing systems usually infer or externalize this knowledge through retrieval, summaries, graphs, rules, or reverse specifications. We investigate a complementary representation in which selected code units directly carry agent-usable knowledge. We introduce Executable Code Knowledge (ECK) and define an Executable Code Knowledge Unit (ECKU) as a source-bound object combining stable identity, semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and a query interface. Our Python prototype supports code-local authoring, manifest export, evidence execution, exact changed-line impact, freshness checking, and agent-facing projections. Across three real Python repositories and 26 controlled patch tasks, direct ECK provides executable test coverage for 11/11 evidence-bearing tasks and exact selectors for 9/11; hiding declared evidence reduces exact recovery to 1/11 (paired exact McNemar p=0.0078). ECK-derived rules recover 11/11 exact selectors, showing that rules are effective delivery artifacts while ECK supplies source binding, validation state, impact, and freshness. Exact changed-line impact matches independently authored labels on all 26 patches (12 unit links; precision, recall, and F1 all 1.000). AST-bounded fingerprints classify 50 positive changes and 17 unrelated same-file controls correctly, whereas static rules snapshots detect none of the 50 stale cases. Model-backed patch-review and cross-layer studies measure projection fidelity rather than independent impact discovery. These results support a hybrid architecture: retrieval for coverage, ECK for source and evidence governance, and projections for delivery.
Figures & tables
| Repository | Direct ECKUs | With Evidence | Controlled Patches |
|---|---|---|---|
| python-dotenv | 6 | 6 | 10 |
| python-slugify | 5 | 4 | 8 |
| tomli | 6 | 6 | 8 |
| Total | 17 | 16 | 26 |
| Method | File Recall | Exact Selector Recall* | Executable Coverage Recall* |
|---|---|---|---|
| Plain | 0.500 | 0.000 | 0.000 |
| BM25 Chunk | 0.981 | 0.455 | 0.455 |
| EKP Retrieval | 0.846 | 0.000 | 0.636 |
| Rules Context | 0.865 | 1.000 | 1.000 |
| Direct ECK without Evidence | 0.981 | 0.091 | 0.636 |
| Direct ECK | 0.981 | 0.818 | 1.000 |
| Method | File Recall | Exact Selector Recall* | Executable Coverage Recall* |
|---|---|---|---|
| Plain | 0.904 | 0.000 | 0.273 |
| EKP Retrieval | 0.904 | 0.091 | 0.636 |
| Direct ECK | 0.962 | 0.727 | 0.727 |
| Metric | Value |
|---|---|
| Direct ECKUs | 17 |
| Direct ECKUs with Evidence | 16 |
| Extractive EKPs | 169 |
| Patches Touched by Direct ECK | 11 |
| Patches Touched by EKP-RAG | 26 |
| Direct Validation Plans | 11 |
| Direct-impact metric | Value |
|---|---|
| True-positive unit links | 12 |
| False-positive unit links | 0 |
| False-negative unit links | 0 |
| Micro precision / recall / F1 | 1.000 / 1.000 / 1.000 |
| Exact-set accuracy, all patches | 1.000 |
| Exact-set accuracy, positive patches | 1.000 |
| Method | Parsed | Strict JSON | Avg Prompt Chars | File Recall | ECK ID Recall | Exact Selector Recall | Executable Coverage Recall |
|---|---|---|---|---|---|---|---|
| Plain Patch | 0.923 | 0.923 | 1344 | 0.923 | 0.000 | 0.000 | 0.273 |
| BM25 Patch | 0.962 | 0.962 | 7443 | 0.962 | 0.591 | 0.364 | 0.636 |
| Rules Patch | 0.923 | 0.923 | 4056 | 0.923 | 0.000 | 0.818 | 0.818 |
| Direct ECK Patch | 1.000 | 1.000 | 3875 | 1.000 | 1.000 | 1.000 | 1.000 |
| Method | ECK ID Recall 95% CI | Exact Selector Recall 95% CI | Executable Coverage Recall 95% CI |
|---|---|---|---|
| BM25 Patch | 0.591 [0.273, 0.864] | 0.364 [0.091, 0.636] | 0.636 [0.364, 0.909] |
| Direct ECK Patch | 1.000 [1.000, 1.000] | 1.000 [1.000, 1.000] | 1.000 [1.000, 1.000] |
| Plain Patch | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] | 0.273 [0.000, 0.545] |
| Rules Patch | 0.000 [0.000, 0.000] | 0.818 [0.545, 1.000] | 0.818 [0.545, 1.000] |
| Method | Avg Prompt Chars | ECK ID Recall | API Recall | DB Field Recall | Business Rule Recall | Exact Selector Recall | Executable Coverage Recall |
|---|---|---|---|---|---|---|---|
| Plain | 1591 | 0.000 | 0.559 | 0.108 | 0.059 | 0.118 | 0.118 |
| BM25 | 9749 | 0.588 | 0.971 | 0.435 | 0.206 | 0.706 | 0.706 |
| Rules | 4947 | 0.000 | 0.971 | 0.617 | 0.471 | 1.000 | 1.000 |
| Rules with IDs | 5300 | 1.000 | 0.971 | 0.631 | 0.382 | 1.000 | 1.000 |
| Direct ECK | 6930 | 1.000 | 0.971 | 0.531 | 0.294 | 1.000 | 1.000 |
| ECK Projection | 4586 | 1.000 | 0.971 | 0.832 | 1.000 | 1.000 | 1.000 |
| Perturbation | Expected | Observed |
|---|---|---|
| source body | stale: source | stale: source |
| agent-facing knowledge | stale: knowledge | stale: knowledge |
| contract | stale: contract | stale: contract |
| evidence | stale: evidence | stale: evidence |
| unchanged | fresh | fresh |
| unrelated edit, same file | fresh | fresh |
| Perturbation | Cases | Correct | Accuracy |
|---|---|---|---|
| source body (positive) | 17 | 17 | 1.000 |
| agent-facing knowledge (positive) | 17 | 17 | 1.000 |
| evidence (positive) | 16 | 16 | 1.000 |
| unrelated same-file edit (negative) | 17 | 17 | 1.000 |
| total | 67 | 67 | 1.000 |
| Capability | Retrieval / Code Search | Rules / Memories | Direct ECK | ECK Projection |
|---|---|---|---|---|
| Broad repository coverage | high | medium | selective | selective |
| Natural agent delivery | medium | high | medium | high |
| Stable code-unit identity | low | possible as generated text | high | high |
| Executable validation evidence | indirect | possible as text | first-class field | first-class field |
| Patch-to-knowledge impact | indirect | low | first-class operation | first-class operation |
| Source/knowledge/evidence freshness | low | low | first-class operation | first-class operation |
| Purpose | Script or Artifact |
|---|---|
| ECK unit discovery, query, validation, freshness, impact, report | eck/*.py |
| Planning experiment | experiments/run_agent_context_experiment.py |
| Patch-review experiment | experiments/run_patch_review_experiment.py |
| Patch-review reanalysis and bootstrap CI | experiments/analyze_patch_review_results.py |
| Rules staleness comparison | experiments/compare_rules_staleness.py |
| Positive/negative freshness controls | experiments/run_eck_freshness_perturbation.py and experiments/run_eck_real_repo_freshness.py |