Producing synthetic data for comparison testing a migrated COBOL application is a major challenge that agentic migrations face today. Our prior paper, the Agentic Method for Deterministic Validation of Legacy Code Migration, presents a deterministic oracle to facilitate such migrations, the Locksmith Loop. We now propose a novel method for achieving high branch and paragraph coverage through generated synthetic end-to-end test cases, by means of entry-replay test-case certification by observing the code execution trace on a surrogate COBOL program that mocks external interactions via a chronological external-operation tape. We applied our method to a very large COBOL program with 33,429 lines of code to generate 849 synthetic end-to-end test cases covering 2,973/3,806 source branches (78.1%) and 989/1,033 main-AST paragraphs (95.7%). Using the Locksmith Loop's projected-agreement validation, the generated Java code matched the COBOL surrogate on the specified observations across all 849 certified synthetic end-to-end test cases. During the campaign we detected 61 defects and fixed the Java generator accordingly. To test whether the coding agent could silently introduce business logic bugs, we seeded 450 defects into the Java code, such as deleted writes, deleted paragraph calls, and altered rounding, to verify how many of these would be detected by the gating methodology. The gate detected 47.6%, including all deleted writes and paragraph calls in executed code, but was unable to detect the rounding and precision defects.
Figures & tables
Fig. 1: The workflow has three noninterchangeable claims: candidate construction, source certification, and target agreement.
Fig. 4: Certified coverage as the catalog grows, measured every 50 witnesses and at 805 and 849.
Fig. 5: Certified branch coverage by source decision construct. The aggregate is 2,973/3,806 outcomes (78.1%) across 849 entry-replay-certified witnesses.
Stage
Runs
Wall clock
Fresh source replay and capture
849
239.826 s
Precompiled Java catalog replay
849
502.846 s
Combined measured validation gate
1,698
742.672 s
TABLE I: Measured runtime of the final 849-witness validation gate. This measurement excludes witness construction, source compilation, Java generation, and Java compilation and is not an end-to-end construction-cost measurement.
Measure
Value
Distinct source branch profiles (not compared)
807
Distinct combined compared projections
514
Distinct final output-dataset images
316
Distinct external-operation key sequences
306
Distinct first-occurrence paragraph traces
322
Distinct normalized display sequences
179
TABLE II: Observed projection signatures and scope across 849 witnesses. Branch profiles are reporting-only.
Fig. 6: Scale overview of the recovered graph: 1,033 main-AST paragraphs and 2,029 resolved transfers. Nodes are arranged by longest-path rank in the strongly connected component condensation; filled nodes are the 989 paragraphs entered by certified witnesses. The dense view shows aggregate structure, not evidence about any individual path.
Fig. 7: Entered and unentered main-AST paragraphs by longest-path rank in the recovered graph’s strongly connected component condensation. Coverage is not a shallow prefix; unentered paragraphs occur throughout the depth.
Fig. 8: (a) Covered source outcomes by number of covering witnesses. (b) Java agreement at each replay checkpoint, ending at 849/849. The dashed line marks catalog growth from 213 to 849 witnesses, not a regression.
Operator
Reached
Killed
Rate
Deleted WRITE / REWRITE
43
43
100%
Deleted PERFORM
40
40
100%
Negated condition
42
39
93%
Deleted assignment/ MOVE
38
26
68%
Relational operator flip
43
22
51%
REDEFINES overlay offset
19
6
32%
TABLE IV: Seeded Java-only mutants, 45 per operator. Reached: a certified witness executes the mutated line (for overlays, enters a paragraph naming the field). The gate also killed 8 unreached REDEFINES mutants (214 of 450 overall).
Fig. 9: Sixty-nine ledgered repair causes by semantic layer; annotations list each bar’s mechanisms from left to right.