Beyond Accuracy: Disentangling Decomposition from Typed-State Quality Assurance in Multi-Agent LLM Pipelines

Maria Gabriela Alves Zuppardo; Vinicius Cardoso Garcia

cs.SEOct 22, 2026

Beyond Accuracy: Disentangling Decomposition from Typed-State Quality Assurance in Multi-Agent LLM Pipelines

Authors: Maria Gabriela Alves Zuppardo, Vinicius Cardoso Garcia

Organizations: Centro de Informática, Universidade Federal de Pernambuco (UFPE), Recife, PE, Brazil

Abstract

Multi-agent LLM pipelines are often credited with gains in software engineering tasks, but it is rarely clear whether a gain stems from the typed contract between agents or from the specialized prompts that decomposition enables. We study this attribution problem in Requirements Engineering, comparing three architectures for classifying requirements as functional or non-functional and then prioritizing them with MoSCoW: a single-call baseline; a two-call variant that reuses the pipeline's specialized prompts but passes an unvalidated intermediate; and a pipeline with a Pydantic-validated typed state. The 18 conditions cover PROMISE NFR+ (625 requirements), three locally hosted small language models, and two languages. The F1 gain over the baseline is associated with the two-call, specialized-prompt variant (up to +0.075), whereas the typed state adds a small difference (-0.003 to +0.013) that is significant in one of six comparisons. Beyond accuracy, the typed pipeline contains out-of-range model outputs at the agent boundary, while the unvalidated variant fails later and drops the requirement, and it shows a small, reproducible increase in inter-model MoSCoW consistency whose cause our design cannot isolate. The study is associative: the contrasts bundle prompts, call count, and staged context, and the results come from three 7--8B models on one dataset. The implementation and replication package are archived on Zenodo (https://doi.org/10.5281/zenodo.23380444).

Figures & tables

Property	Baseline	Two-call	Pipeline
LLM calls per requirement	1	2	2
Prompts	joint	task-specific	task-specific (same as two-call)
Intermediate result	none	plain dataclass	validated Pydantic object
First Pydantic validation	after the single call	after the second call	after the first call (agent boundary)
Confidence given to the prioritiser	—	fixed at 1.0	passed; below 0.70 the prompt asks for conservative labels
Orchestration	direct agent call	direct agent call	LangGraph StateGraph

Table 1. Differences between the three architectures. Shared by all: model, temperature, workers, retry policy, dataset adapter, response parsing, and metrics. EN conditions use the original English text and an English prompt; PT conditions use the translated text and a Portuguese prompt.

Model	Params	Quant.
qwen2.5:7b	7.6B	Q4_K_M
llama3.1:8b	8.0B	Q4_K_M
mistral:7b	7.2B	Q4_K_M

Table 2. Models evaluated. All use Q4_K_M quantization.

Factor	Levels
Architecture	baseline, pipeline, two-call baseline
Model	qwen2.5:7b, llama3.1:8b, mistral:7b
Language	EN (original), PT (machine-translated)

Table 3. Experimental grid (18 conditions).

Model	L	Acc	F1	MCC	Acc	F1	MCC	$\Delta$ F1	$h$	$p_{W}$	$p_{M}$	$g$	Sig.
		Baseline			Pipeline
qwen2.5:7b	PT	0.733	0.733	0.527	0.814	0.812	0.638	+0.079	+0.193	$3.3\times 10^{-7}$	$6.4\times 10^{-7}$	+0.253	✓
qwen2.5:7b	EN	0.784	0.784	0.613	0.808	0.807	0.639	+0.023	+0.060	0.059	0.077	+0.119 †	–
llama3.1:8b	PT	0.640	0.630	0.446	0.696	0.693	0.511	+0.063	+0.119	$7.4\times 10^{-6}$	$7.7\times 10^{-6}$	+0.287	✓
llama3.1:8b	EN	0.728	0.724	0.557	0.730	0.728	0.556	+0.004	$-$ 0.011	0.680	0.0073	+0.154 †	$\checkmark^{\ddagger}$
mistral:7b	PT	0.602	0.586	0.380	0.579	0.560	0.345	$-$ 0.026	$-$ 0.046	0.071	0.092	$-$ 0.117 †	–

Table 4. RQ1, pipeline vs. baseline (primary metric: macro F1). Acc, F1 and MCC exclude confidence-zero parse-failure fallbacks (§ 6.6 ).

p_{W}

: Wilcoxon signed-rank on the same data.

p_{M}

: exact McNemar on 623 pairs retaining the fallbacks (Bonferroni

\alpha^{\prime}=0.0083

\checkmark

: significant in per-requirement correctness in the primary analysis (fallbacks excluded,

p_{W}<\alpha^{\prime}

); it does not mark the macro-F1 bootstrap of Table 5 .

\ddagger

: significant only in the correctness test that retains fallbacks (

p_{M}<\alpha^{\prime}

g

: paired Cohen’s

g

;

\dagger

: interval spans several magnitude bands, not reliably estimable.

Model	L	$\Delta$ [adj. interval]	$n$	$\Delta$ [adj. interval]	$n$	$\Delta$ [adj. interval]	$n$
		Pipeline $-$ Baseline		$\Delta_{\text{p}}$ (2-call $-$ Base)		$\Delta_{\text{s}}$ (Pipe $-$ 2-call)
qwen2.5:7b	PT	+0.079 [+0.042, +0.119]	621	+0.076 [+0.034, +0.115]	623	+0.005 [ $-$ 0.011, +0.020]	621
qwen2.5:7b	EN	+0.023 [ $-$ 0.009, +0.057]	623	+0.020 [ $-$ 0.013, +0.054]	623	+0.003 [ $-$ 0.005, +0.013]	623
llama3.1:8b	PT	+0.063 [+0.028, +0.100]	623	+0.053 [+0.018, +0.087]	623	+0.011 [ $-$ 0.003, +0.026]	623
llama3.1:8b	EN	$-$ 0.005 [ $-$ 0.041, +0.028]	590	$-$ 0.003 [ $-$ 0.036, +0.029]	590	$-$ 0.002 [ $-$ 0.013, +0.010]	623
mistral:7b	PT	$-$ 0.027 [ $-$ 0.066, +0.011]	623	$-$ 0.024 [ $-$ 0.062, +0.015]	623	$-$ 0.003 [ $-$ 0.014, +0.006]	623

Table 5. Paired, class-stratified bootstrap of macro-F1 differences (

B=10{,}000

). Shown: point estimate and Bonferroni-adjusted interval (

\alpha^{\prime}=0.05/6

, i.e. a 99.17% percentile interval); bold when that interval excludes zero.

n

: requirements valid (not a fallback) for both sides of that contrast. Point estimates can differ slightly from Tables 4 and 9 , which compute each F1 on its own valid set. Unadjusted 95% intervals are in macro_f1_bootstrap.json .

Arch.	Model	PT	EN	$h$	$p_{MW}$	$p_{M}$
Baseline	qwen2.5:7b	0.734	0.785	$-$ 0.120	0.034	$3.0\times 10^{-5}$
Baseline	llama3.1:8b	0.640	0.729	$-$ 0.191	$<0.001$	0.0009
Baseline	mistral:7b	0.602	0.607	$-$ 0.010	0.862	0.801
Pipeline	qwen2.5:7b	0.815	0.809	+0.015	0.793	0.913
Pipeline	llama3.1:8b	0.697	0.730	$-$ 0.075	0.188	0.011
Pipeline	mistral:7b	0.579	0.638	$-$ 0.121	0.034	$9.1\times 10^{-6}$

Table 6. Language-effect results.

p_{MW}

: Mann-Whitney U (unpaired, original analysis; fallbacks excluded).

p_{M}

: exact McNemar on 623 PT/EN-aligned pairs (fallbacks retained); bold = significant after Bonferroni over 9 comparisons (

\alpha^{\prime}=0.0056

). Accuracies follow the exclusion policy of § 6 ; four of the 18 cells are affected (§ 6.6 ).

Architecture	L	$n$	$\kappa$	Interp.
Baseline	PT	623	$-$ 0.1406	poor
Baseline	EN	623	$-$ 0.1220	poor
Two-call	PT	623	+0.1360	slight
Two-call	EN	623	+0.1695	slight
Pipeline	PT	623	+0.1768	slight
Pipeline	EN	623	+0.2388	fair

Table 7. Inter-model MoSCoW consistency, Fleiss’

\kappa

(3 raters, 6 strata), full four-label priority set.

n

: requirements aligned across all three raters (same filter as RQ1).

Contrast	What changes	What it supports
$\Delta_{\text{p}}$ : baseline $\to$ two-call	calls, task prompts, staged context	F1 associated with the variant (not which element)
$\Delta_{\text{s}}$ : two-call $\to$ pipeline	validation boundary, confidence, LangGraph	bound on the F1 difference; QA properties (not their cause)

Table 8. What each contrast changes and permits us to conclude.

Model	L	Base	2Call	Pipe	$\Delta_{p}$	$\Delta_{s}$	$\Delta_{p}$	$\Delta_{s}$	$b/c$	$g$
					Excluded		Retained
qwen2.5:7b	PT	0.733	0.808	0.812	+0.075	+0.004	+0.075	+0.001	8/7	+0.033 †
qwen2.5:7b	EN	0.784	0.804	0.807	+0.020	+0.003	+0.020	+0.003	3/1	+0.250 †
llama3.1:8b	PT	0.630	0.682	0.693	+0.053	+0.011	+0.053	+0.011	8/2	+0.300 †
llama3.1:8b	EN	0.724	0.730	0.728	+0.006	$-$ 0.002	+0.045	$-$ 0.002	3/4	$-$ 0.071 †
mistral:7b	PT	0.586	0.563	0.560	$-$ 0.023	$-$ 0.003	$-$ 0.023	$-$ 0.003	1/3	$-$ 0.250 †

Table 9. Decomposition ablation (macro F1).

\Delta_{p}

= F1(two-call)

-

F1(baseline), a difference associated with the two-call variant (prompt wording, extra call, and staged context are confounded);

\Delta_{s}

= F1(pipeline)

-

F1(two-call). Excluded drops the confidence-zero fallbacks (§ 6.6 ); Retained keeps them as errors (

n=625

;

\Delta

computed from unrounded F1).

b/c

: requirements correct only for the pipeline / only for the two-call baseline.

g

: paired Cohen’s

g

for

\Delta_{s}

;

\dagger

: not reliably estimable (4–15 discordant pairs).

Architecture	Value-range (6)	Parse-level (7)
Baseline	flagged, 1 call	flagged, 1 call
Two-call	dropped, 6 calls	flagged, 2 calls
Pipeline	flagged, 2 calls	flagged, 2 calls

Table 10. Fault-injection campaign: outcome and LLM calls consumed per fault (13 faults: 6 value-range, 7 parse-level).

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,050 papers

Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance
May 12, 2026Elias CalboreanuMulti-Agent LLM Systems AI Agent Auditing
Swift AI Lab, The Swift Group, LLC, Reston, VA 20190, USA
Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models
Jul 29, 2026Ashish Prajapati, Om MohiteMulti-Agent LLM Systems LLM Self-Refinement
Independent Researcher Mumbai, India
Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows
Oct 2, 2026Merve Astekin, Yan Naing Tun, Arda Goknil +5Software Engineering Agents LLM Agent Evaluation
SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye

May 12, 2026J/K move · Enter open · S save

Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance

Elias Calboreanu

Swift AI Lab, The Swift Group, LLC, Reston, VA 20190, USA

Multi-Agent LLM Systems AI Agent Auditing LLM Auditing

Open paper AskPDF arXiv

Prompt specifications for multi-agent large language model (LLM) systems carry data contracts and integration logic across interdependent files but are rarely subjected to structured-inspection rigor. We report a single-system case study of iterative, agent-driven auditing applied to AEGIS (Autonomous Engineering Governance and Intelligence System), a seven-lane production pipeline whose 7152-line specification surface was audited across nine rounds, surfacing 51 consistency defects (per-round counts of 15, 8, 12, 2, 8, 1, 4, 1, 0). We present a seven-category post hoc taxonomy with explicit coding rules, non-monotonic convergence consistent with cascading edits and audit-scope expansion, and a locked audit protocol. We further report two partial replications on a public synthetic mini-specification: a cross-LLM panel of four frontier vendors (OpenAI, Anthropic, Google, xAI; 12 traces; multi-vendor union detects all five seeded defects) and an inter-rater reliability check on a stratified subsample (Cohen's

κ

= 0.80 on category, 0.46 on severity). The full reproducibility bundle accompanies the submission.

Abstract

Multi-Agent LLM Systems

Explore similar work

Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance

Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows