Reusing Assurance Evidence in AI Pipelines: Validity Conditions, a Fail-Closed Release Gate, and a Replay of 714 Public Data Version Control Pipelines
Organizations: Independent Researcher, New Jersey, USA · Independent Researcher, USA
Abstract
Rerunning every governance control after every change to an artificial intelligence (AI) pipeline is expensive, so teams reuse earlier results. Reuse is safe only if each control's declared dependencies cover everything that can change its outcome, and only if the release gate checks that reused evidence describes the candidate being released. This article makes both conditions explicit and measures how often the first fails in practice. It defines when a signed evidence record is valid for a candidate and gives sufficient conditions under which change-local re-execution reaches the same release decision as rerunning every control. A reference gate implementing these checks with Ed25519 digital signatures blocks eight of nine manipulation scenarios, against one for a gate that trusts the planner and three for a gate that only verifies signatures; a dependency the policy never declared passes every gate. A replay of 38,083 commits from 714 public Data Version Control (DVC) pipelines measures that failure mode. Reusing stage results according to declared dependencies avoids 83.4% of stage executions, but in 3.3% of commits it reuses a stage whose imported code or referenced configuration changed, and 31% of pipelines are affected. A manual audit confirms the undeclared dependency in 36 of 40 sampled events. Evaluation stages, whose outputs are the evidence, are the most exposed. Adding each stage's static import closure lowers stale reuse to 0.4% while still avoiding 81.6% of executions; sampled shadow re-execution removes far less, and rerunning everything after unexplained changes avoids only 45.3%.
Figures & tables
| Line of work | Checkable evidence | Reuse validity conditions | Independent fail-closed gate | Measures declaration completeness | Domain |
|---|---|---|---|---|---|
| Executable governance [ 3 , 4 , 5 , 6 ] | Yes | No | Yes | No | AI governance |
| Supply-chain attestation [ 17 , 18 , 19 ] | Yes | Partly | Yes | No | Software artifacts |
| Build and test dependency studies [ 23 , 24 , 25 ] | No | Yes | No | Yes | Builds |
| DVC co-evolution [ 32 ] | No | No | No | No | ML pipelines |
| This article | Yes | Yes | Yes | Yes | ML pipelines |
| Step | Repositories |
| Candidates from search | 3,524 |
| Root dvc.yaml on default branch | 2,130 |
| Cloned successfully | 2,087 |
| At least two stages at HEAD | 1,880 |
| At least one resolvable Python stage | 1,644 |
| At least 10 analyzable commits | 726 |
| Scenario | Planner-trusting | Signature-only | Full gate | |
|---|---|---|---|---|
| S0 | Documentation-only change. Only provenance is invalidated and re-run; other evidence is reused. | R ∗ | R ∗ | R ∗ |
| S1 | Dataset change, correct re-run. Invalidated controls re-run and pass. | R ∗ | R ∗ | R ∗ |
| S2 | Planner omits an invalidated control. Fairness evidence for the old dataset is presented. | R | R | H ∗ |
| S3 | Cross-context replay. Evidence certified for a different deployment is reused. | R | R | H ∗ |
| S4 | Tampered evidence record. A failing privacy result is edited to pass after signing. | R | H ∗ | H ∗ |
| S5 | Self-certified result. The model team signs its own robustness pass. | R | R | H ∗ |
| Subset | Repositories | Commits | Declared: executed (%) | Declared: stale reuse (%) | Declared + imports: stale reuse (%) |
|---|---|---|---|---|---|
| All repositories | 714 | 38,083 | 16.6 | 3.3 [2.4, 4.5] | 0.4 |
| At least 50 analyzed commits | 145 | 25,992 | 16.6 | 3.6 [2.3, 5.4] | 0.4 |
| At least 5 stages at HEAD | 330 | 19,355 | 13.4 | 4.8 [3.3, 6.7] | 0.5 |
| At least 5 GitHub stars | 22 | 5,343 | 46.4 | 3.0 [0.6, 10.3] | 0.1 |
| Last commit in 2026 | 548 | 28,696 | 12.5 | 3.0 [2.0, 4.2] | 0.3 |
| Largest 1% of repositories removed | 707 | 27,768 | 15.3 | 4.2 [3.2, 5.2] | 0.5 |
| Rule | Executed (%) | Stale reuse, code/config (%) | Stale reuse, incl. environment (%) |
|---|---|---|---|
| Full | 100.0 | 0.0 | 0.0 |
| Declared (DVC) | 16.6 [11.4, 25.0] | 3.3 [2.4, 4.5] | 13.2 |
| Declared + imports | 18.4 [12.8, 27.0] | 0.4 [0.3, 0.7] | 10.5 |
| Declared + imports + environment | 26.2 [19.8, 35.3] | 0.4 [0.2, 0.6] | 0.4 |
| Fail-closed fallback | 54.7 [48.6, 62.2] | 0.4 [0.2, 0.6] | 0.4 |
| Reference (code/config) | 18.7 | 0.0 | — |
| Label | Events | Share (%) |
|---|---|---|
| Material: real dependency, output plausibly changed | 14 | 35 |
| Benign: real dependency, behavior-neutral change | 22 | 55 |
| Analyzer error | 4 | 10 |
| Stage role | Stage slots | Stale (% of reused) | Share of all stale (%) |
| Data preparation | 75,793 | 2.0 | 32 |
| Training | 33,689 | 2.8 | 18 |
| Evaluation | 31,043 | 4.9 | 29 |
| Other | 68,342 | 1.7 | 21 |
| Commit touched | Commits | Stale reuse (% of commits) | |
| Code | 13,047 | 8.9 |