Orchestrating Human-AI Software Delivery: A Retrospective Longitudinal Field Study of Three Software Modernization Programs

Maximiliano Armesto; Christophe Kolb

cs.SEMar 20, 2026

Orchestrating Human-AI Software Delivery: A Retrospective Longitudinal Field Study of Three Software Modernization Programs

Authors: Maximiliano Armesto, Christophe Kolb

Organizations: Taller Technologies

Abstract

Evidence on AI in software engineering still leans heavily toward individual task completion, while evidence on team-level delivery remains scarce. We report a retrospective longitudinal field study of Chiron, an industrial platform that coordinates humans and AI agents across four delivery stages: analysis, planning, implementation, and validation. The study covers three real software modernization programs -- a COBOL banking migration (~30k LOC), a large accounting modernization (~400k LOC), and a .NET/Angular mortgage modernization (~30k LOC) -- observed across five delivery configurations: a traditional baseline and four successive platform versions (V1--V4). The benchmark separates observed outcomes (stage durations, task volumes, validation-stage issues, first-release coverage) from modeled outcomes (person-days and senior-equivalent effort under explicit staffing scenarios). Under baseline staffing assumptions, portfolio totals move from 36.0 to 9.3 summed project-weeks; modeled raw effort falls from 1080.0 to 232.5 person-days; modeled senior-equivalent effort falls from 1080.0 to 139.5 SEE-days; validation-stage issue load falls from 8.03 to 2.09 issues per 100 tasks; and first-release coverage rises from 77.0% to 90.5%. V3 and V4 add acceptance-criteria validation, repository-native review, and hybrid human-agent execution, simultaneously improving speed, coverage, and issue load. The evidence supports a central thesis: the largest gains appear when AI is embedded in an orchestrated workflow rather than deployed as an isolated coding assistant.

Figures & tables

Figure 1: Orchestrated agentic software delivery across analysis, planning, implementation, and validation stages, showing hybrid human–AI coordination.

Version	Dominant capabilities	Practical interpretation
V1	Tool-centric agent usage for repository analysis, migration documentation, backlog generation, and autonomous task execution, with limited integrated validation or repository-native review.	Earliest gains appear in analysis and planning, but downstream quality weakens.
V2	CLI-orchestrated delivery pipelines for context preparation, implementation, and validation.	Architectural normalization yields only modest improvement over V1.
V3	Shared web workspace, brainstorming interface, task board, automatic task pickup, and first-generation validation against task acceptance criteria.	Implementation and validation improve sharply once orchestration becomes task-centric.
V4	Repository authentication, branch and PR workflows, code review, document ingestion, process monitoring, and hybrid human-agent execution.	Best observed balance of speed, first-release coverage, and downstream issue load.

Table 1: Versioned delivery configurations.

Project	Approx. size	Modernization summary	Traditional baseline
Bank application	$\sim$ 30k LOC	COBOL backend to Python, terminal frontend to Next.js, and database migration to PostgreSQL.	10.0 weeks
ACAS accounting platform	$\sim$ 400k LOC	Large legacy accounting modernization.	20.0 weeks
Mortgage application	$\sim$ 30k LOC	.NET 3 to .NET 8, Angular to React, and REST to GraphQL.	6.0 weeks

Table 2: Benchmark portfolio.

Version	Weeks	Raw person-days	SEE-days	Issue load / 100 tasks	Task-weighted coverage
Traditional	36.0	1080.0	1080.0	8.03	77.0%
V1	28.6	715.0	429.0	8.63	52.6%
V2	26.4	660.0	396.0	8.17	57.6%
V3	13.8	345.0	207.0	4.19	83.4%
V4	9.3	232.5	139.5	2.09	90.5%

Table 3: Portfolio evolution across delivery configurations. Coverage values are recalculated directly from the project tables and task weights.

Contrast	Speedup	Time red.	Raw-effort red.	SEE red.	Issue-load red.	Coverage change
Traditional $\rightarrow$ V4	3.87 $\times$	74.2%	78.5%	87.1%	74.0%	+13.4 pp
V1 $\rightarrow$ V4	3.08 $\times$	67.5%	67.5%	67.5%	75.8%	+37.9 pp

Table 4: Two informative comparisons for the mature V4 configuration. The V1-to-V4 contrast is especially useful because both configurations share the same nominal agentic staffing scenario.

Figure 2: Total measured duration normalized to each project’s traditional baseline. Values below 1 indicate faster delivery.

Figure 3: First-release coverage and validation-stage issue load across delivery configurations. The issue metric should be interpreted as a workload-normalized downstream escape measure rather than intrinsic defect density.

Figure 4: V4 stage durations as fractions of the traditional baseline. Lower values indicate shorter duration in the measured stage.

Junior weight $w$	Agentic SEE staffing	V4 SEE-days	Reduction vs. traditional
0.25	2.0	93.0	91.4%
0.50	3.0	139.5	87.1%
0.75	4.0	186.0	82.8%
1.00	5.0	232.5	78.5%

Table 5: Sensitivity of modeled V4 senior-equivalent effort to alternative junior weights. The traditional baseline remains fixed at 6 SEE.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Version	Analysis	Planning	Impl.	Val.	Total	Tasks	Issues	Coverage
Traditional	2.0	2.0	4.0	2.0	10.0	400	35	80%
V1	1.0	0.4	3.0	3.0	7.4	800	80	55%
V2	0.8	0.4	2.5	3.0	6.7	800	76	60%
V3	0.6	0.3	1.5	1.5	3.9	800	40	85%
V4	0.2	0.2	1.0	1.0	2.4	800	20	90%

Appendix

Table 6: Bank application benchmark across delivery configurations.

Version	Analysis	Planning	Impl.	Val.	Total	Tasks	Issues	Coverage
Traditional	5.0	3.0	8.0	4.0	20.0	2500	200	75%
V1	2.5	1.0	6.0	7.0	16.5	4500	380	50%
V2	2.2	0.9	5.2	7.0	15.3	4500	360	55%
V3	1.5	0.6	3.2	2.0	7.3	4500	180	82%
V4	0.8	0.6	2.0	1.5	4.9	4500	90	90%

Appendix

Table 7: ACAS accounting benchmark across delivery configurations.

Version	Analysis	Planning	Impl.	Val.	Total	Tasks	Issues	Coverage
Traditional	1.5	1.0	2.0	1.5	6.0	300	22	90%
V1	0.8	0.4	1.5	2.0	4.7	550	45	70%
V2	0.7	0.4	1.3	2.0	4.4	550	42	75%
V3	0.4	0.2	1.0	1.0	2.6	550	25	92%
V4	0.2	0.2	0.8	0.8	2.0	550	12	95%

Appendix

Table 8: Mortgage application benchmark across delivery configurations.

Project	Version	Raw person-days	SEE-days	Issue load / 100 tasks
Bank	Traditional	300.0	300.0	8.75
Bank	V1	185.0	111.0	10.00
Bank	V2	167.5	100.5	9.50
Bank	V3	97.5	58.5	5.00
Bank	V4	60.0	36.0	2.50
ACAS	Traditional	600.0	600.0	8.00

Appendix

Table 9: Modeled effort and observed validation-stage issue load by project and version.

Metric	Type	Definition	Interpretation caveat
Total duration $T_{p,v}$	Observed	Sum of measured analysis, planning, implementation, and validation durations.	Excludes deployment because that stage was not represented consistently across versions.
Task count $N_{p,v}$	Observed	Number of recorded work items in the project backlog for the project-version cell.	Task granularity changes materially across delivery configurations.
Validation issues $I_{p,v}$	Observed	Issues reaching downstream validation.	Captures defect escape to validation, not total defects created anywhere in the pipeline.
Validation-stage issue load $L_{p,v}$	Derived from observed measures	$100\times I_{p,v}/N_{p,v}$ , reported as issues per 100 tasks.	A workload-normalized escape metric; not equivalent to intrinsic defect density because the denominator is intervention-sensitive.
First-release coverage $C_{p,v}$	Observed	Fraction of first-release requirements recorded as completed and accepted at initial handoff.	Sensitive to requirement scoping and adjudication practices.
Raw person-days $E^{\mathrm{raw}}_{p,v}$	Modeled	$5h_{v}T_{p,v}$ under full-time staffing with $h_{\mathrm{Traditional}}=6$ and $h_{\mathrm{Agentic}}=5$ .	A scenario-based effort estimate, not directly logged labor-hours.

Appendix

Table 10: Metric taxonomy used in the paper.

Subset	Duration ratio	Issue-load ratio	Coverage change
All projects	0.258	0.260	+13.4 pp
Exclude Bank	0.265	0.255	+13.9 pp
Exclude ACAS	0.275	0.291	+7.8 pp
Exclude Mortgage	0.243	0.256	+14.3 pp

Appendix

Table 11: Leave-one-project-out sensitivity for the V4 versus traditional portfolio comparison. Duration ratio is V4 divided by traditional, and issue-load ratio is the same ratio for validation-stage issue load.

Project	Pre-review issues	Validation issues	Containment rate
Bank	40	20	50.0%
ACAS	188	90	52.1%
Mortgage	23	12	47.8%

Appendix

Table 12: Approximate V4 review containment. Pre-review issues are the benchmark’s recorded issues observed before downstream validation.

Explore similar work

Jul 6, 2026

Three-Phase Evaluation of AI-Assisted Software Development Life Cycle

This paper presents an exploratory evaluation of how increasing levels of AI autonomy affect software development productivity, requirement adherence, and developer cognitive workload. A team of four developers reimplemented the same full-stack web application across three sequential phases: partial AI-assisted development using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro. Evaluation metrics included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures. Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload, while developer frustration increased modestly. The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.

Joshua Strubel, Professor Carrie Russell, Carson Crockett +4

Dr. Joshua Strübel, Professor Carrie Russell, Carson Crockett, Jason Ferraro, Nathan Londhe, Uzayr Syed, and Jacob Viehe

Human-AI Collaboration AI-Assisted Software Development

Ask PDF

Jun 4, 2026

Human Oversight and Overload: Two Hidden and Costly Burdens of AI-Assisted Software Engineering

AI is changing how software engineers work, but it often comes with hidden burdens and costs. In this paper, we characterize two such often-overlooked burdens: (1) the constant need for human oversight and inspection of AI-generated artifacts; and (2) the growing cognitive overload on software engineers from receiving large amounts of suggestions from AI tools. The need for human oversight is not optional-engineers must review, validate, and sometimes rework what AI produces. At the same time, the flood of AI suggestions, prompts, and possible solutions can leave developers mentally stretched. By blending evidence from recent opinions from practitioners, we highlight these often-overlooked challenges and open a conversation about how teams can handle them in day-to-day AI-assisted software engineering.

Vahid Garousi

Queen’s University Belfast, UK · Azerbaijan Technical University, Azerbaijan

Human Oversight of AI AI-Assisted Software Development

Ask PDF

Jun 2, 2026

SPOQ: Specialist Orchestrated Queuing for Multi-Agent Software Engineering

Multi-agent AI systems show promise for automating software engineering tasks, yet existing approaches suffer from coordination overhead, quality control gaps, and limited human oversight. We introduce SPOQ (Specialist Orchestrated Queuing), a methodology combining three innovations: (1) wave-based topological dispatch that computes parallel execution waves from task dependency graphs; (2) dual validation gates applying quality metrics before execution (planning validation) and after (code validation) to reduce rework cycles; and (3) Human-as-an-Agent (HaaA) integration, where a human specialist participates in decomposition and can be consulted during execution. SPOQ uses a three-tier agent hierarchy (Opus workers, Sonnet reviewers, Haiku investigators) to optimize cost-quality tradeoffs. We evaluate SPOQ through four experiments. Experiment 1: wave dispatch approaches the critical-path lower bound (ratio 1.03--1.11, speedup up to 14.3x); on a 2-slot local backend it delivers a stable 1.4x speedup. Experiment 2: SPOQ improves planning coverage from 93.0 to 99.75, eliminates cyclic plans, and lifts parallelism from 31.0 to 75.25. Experiment 3: dual validation reduces defects from 0.34 to 0.20 per task and lifts test pass rate from 91.25% to 99.75%. Experiment 4: human review reduces residual defects from 0.47 to 0.03 per task. Results are replicated on a locally hosted open-weights model (Qwen3.6-35B-A3B), verifying gains are attributable to orchestration rather than any specific model. A longitudinal study across 17 repositories, 8,589 commits, 1,822 tasks, and 13,866 tests (99.87% pass rate) provides ecological validation.

Royce Carbowitz, Dheeraj Kumar

Pinpoint Technologies LLC

Multi-Agent System Optimization Software Engineering Agents Multi-Agent Orchestration Human-in-the-Loop AI Multi-Agent Planning

Ask PDF