VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

Gabriel Dautovi; Jie JW Wu

cs.SEOct 21, 2026

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

Authors: Gabriel Dautovi, Jie JW Wu

Organizations: Department of Computer Science, Michigan Technological University, Houghton, Michigan, United States

Abstract

Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.

Figures & tables

Figure 1: The V-Model. The left arm decomposes stakeholder needs into progressively more detailed designs, and the right arm integrates and assesses the result in reverse order. Each dashed link pairs a design activity with the test activity that verifies it at the same level of abstraction

Match state	Build half	Verify half	Meaning
matched	present	present	Evidence of the work and of the corresponding check is present
build-only	present	absent	Evidence of the work is present but evidence of the corresponding check is not
verify-only	absent	present	Evidence of a check is present without evidence of corresponding work at this level
unverified	claimed	claimed	The model claimed half was present but cited no evidence found in the pull request
not applicable	–	–	Neither half is relevant to the change

Table 1: Match states of a build and verify pair. Build-only is the state the study treats as leaving the pair open. The unverified state arises only on the three model-judged pairs (Section 3.4)

Build half	Verify half	The question the pair asks	Decided by
Requirement	Acceptance	Is a requirement stated, and is there evidence it was accepted?	model, constrained by signals
System Design	System Test	Did system level behavior change, and is it exercised end to end?	model, constrained by signals
Architecture	Integration	Did a boundary or interface change, and is the integration exercised?	model, constrained by signals
Module Design	Unit Test	Did module logic change, and did the unit test arrive with it?	deterministic rule

Table 2: The four build and verify pairs scored by the instrument. The fourth pair is decided by a deterministic rule over the diff rather than by the model.

Figure 2: The V-Model reduced to the claim this paper tests. Each pair joins a build activity to the verification activity at the same level. The fourth pair is decided by a deterministic rule over the diff. Coding, at the apex, has no mirror half and is therefore not a pair

Figure 3: The architecture of VeeReview. The top row is the path one pull request takes, in order of execution. Deterministic signals reach the rule override directly, so the model cannot overturn a fact established by rule. The bottom row is the run infrastructure that keeps a corpus of verdicts comparable

Figure 4: From corpus to verdicts. Agent-authored pull requests are drawn from the AIDev curated subset, matched to human-authored pull requests within the same repository, and both cohorts are scored by one judge model under one configuration. Each stage is caught and resumable, and the sample and the dataset revision are frozen

Matched pairs	1,454	1,454
	Agent cohort	Human cohort
Repositories	741	741
Median lines changed	84	65
Mean lines changed	1,259	466
Links an issue	29.7%	11.6%
Ships tests	32.5%	23.7%

Table 3: Composition of the matched sample. The two cohorts are matched one to one within repository on size band and a window of

\pm

90 days, so the repository count is shared by construction. The artifact rows and the not-applicable rates are what matching does not equalize.

	R1	R2	Adjudicated
Requirement $\leftrightarrow$ Acceptance (%)	8	28	4
System Design $\leftrightarrow$ System Test (%)	8	44	32
Architecture $\leftrightarrow$ Integration (%)	48	72	60
Module Design $\leftrightarrow$ Unit Test (%)	32	52	44
Wrongly withheld matched (labels)	48	16	30
Wrongly awarded matched (labels)	12	9	11

Table 4: Calibration against the 25-pull-request reference set. The first four rows give the instrument’s agreement with each reference, per pair, as a percentage of 25 labels. The preregistered bar is 80%. The remaining rows give the direction of the instrument’s errors and how they fall across the two cohorts. R1 and R2 are the two raters’ own labels. All differences are agent minus human, in percentage points. Cohort drift is the amount the instrument’s errors add to the agent cohort’s matched rate relative to the human cohort’s. The rater-reported gap is the cohort difference in the reference labels themselves.

Pair	Gap (pp)	95% CI	Discordant a/h	Agent share	p
Requirement $\leftrightarrow$ Acceptance	+16.8	[+13.3, +20.2]	208/55	79.1%	<10 -9
System Design $\leftrightarrow$ System Test	+18.0	[+14.1, +22.1]	162/44	78.6%	<10 -9
Architecture $\leftrightarrow$ Integration	+16.3	[+12.4, +20.1]	101/30	77.1%	<10 -9
Module Design $\leftrightarrow$ Unit Test a	+6.1	[+2.4, +9.8]	168/114	59.6%	0.0016

Table 5: RQ1: matched-rate difference (agent

-

human) per pair. The interval is a 95% bootstrap over repositories computed on the difference itself. "Discordant a/h" counts pairs in which exactly one side matched, and the p value is from an exact two-sided sign test conditioned on those pairs. a Decided by rule, not by the model

Figure 5: RQ1: the full match state distribution per pair and cohort over all 1,454 pairs on each side, so that the not-applicable shares are comparable. Match state is encoded by hue, hatching and a fixed best-to-worst ordering, so the figure survives greyscale reproduction

Pair	Cohort	matched	unverified	verify-only	build-only	na
Requirement $\leftrightarrow$ Acceptance	agent	21.6	4.1	2.7	52.2	19.4
	human	7.2	9.6	1.9	53.2	28.1
System Design $\leftrightarrow$ System Test	agent	21.7	5.3	1.4	37.3	34.4
	human	10.0	9.6	0.4	47.0	32.9
Architecture $\leftrightarrow$ Integration	agent	15.1	5.2	0.7	32.9	46.1
	human	5.8	11.6	0.4	32.3	49.9

Table 6: RQ1: the full five-state distribution per pair and cohort, as a percentage of all 1,454 pairs. Percentages are over every pair, not-applicable included, because the cohorts do not draw not-applicable equally often.

Pair	Agent	Human	Crude gap	Standardized gap	Retained
Requirement $\leftrightarrow$ Acceptance	26.8	10.0	+16.8	+6.8	40%
System Design $\leftrightarrow$ System Test	33.0	15.0	+18.0	+10.7	59%
Architecture $\leftrightarrow$ Integration	28.0	11.7	+16.3	+10.8	66%
Module Design $\leftrightarrow$ Unit Test a	57.7	51.6	+6.1	-1.2	—

Table 7: RQ1: the matched-rate difference before and after direct standardization to the pooled mix of (links an issue)

\times

(ships tests)

\times

(size band). Rates and differences are in percentage points. a Decided by rule, not by the model

Pair	Crude	Standardized, issue $\times$ tests $\times$ size	Standardized, issue $\times$ size	Retained
Requirement $\leftrightarrow$ Acceptance	+16.8	+6.8 [+3.9, +10.2]	+7.2 [+4.2, +10.3]	40% / 43%
System Design $\leftrightarrow$ System Test	+18.0	+10.7 [+7.2, +14.7]	+12.7 [+8.7, +17.0]	59% / 71%
Architecture $\leftrightarrow$ Integration	+16.3	+10.8 [+8.2, +14.5]	+13.0 [+9.4, +17.0]	66% / 79%
Module Design $\leftrightarrow$ Unit Test a	+6.1	$-$ 1.2 [ $-$ 3.9, +2.4]	+3.1 [ $-$ 0.4, +7.1]	— / 51%

Table 8: RQ1: the matched-rate difference after direct standardization under two stratum definitions. Differences are in percentage points, with 95% repository-level bootstrap intervals computed on the standardized difference. The second definition drops test shipping from the stratum because it is conceptually adjacent to the Module Design

\leftrightarrow

Unit Test outcome. Retained is the standardized difference as a share of the crude one, under each definition in turn. a Decided by rule, not by the model

Figure 6: RQ1 sensitivity to matching quality: the difference computed separately over pairs matched inside the agent pull request’s own size band and over pairs that fell back to the nearest match in log size

Figure 7: RQ1 by size band: the cohort difference inside each size band. Circled points are cells below n = 30 on either side, which are reported but not interpreted

Pair	build-only %	95% CI	n	Rank	Rank held
Requirement $\leftrightarrow$ Acceptance	71.6	[68.1, 74.3]	2,872	1	100.0%
System Design $\leftrightarrow$ System Test	50.9	[38.7, 64.5]	2,468	3	78.9%
Architecture $\leftrightarrow$ Integration	57.4	[53.0, 64.5]	2,002	2	78.9%
Module Design $\leftrightarrow$ Unit Test a	37.2	[27.0, 49.6]	2,644	4	100.0%

Table 9: Table 9 RQ2: build-only rate per pair over the agent corpus . Rates are computed over assessed rows (not-applicable excluded), and the interval is a bootstrap over repositories. "Rank held" is the share of bootstrap resamples in which the pair keeps its observed rank. a Decided by rule, not by the model

Figure 8: Fig. 8 RQ2: build-only rate per pair and authoring agent, unweighted. Each agent carries a hatch pattern as well as a color, so the figure survives greyscale reproduction. Bar labels give cell n, and a starred label marks a cell below n = 30, which is reported but not interpreted

PR size (lines changed)	matched %	build-only %	na %	n rows	truncated
XS (<50)	13.3	36.1	45.8	5,844	0.0%
S (50-200)	27.2	39.6	24.6	4,164	0.0%
M (200-1k)	34.3	38.3	19.5	3,004	0.4%
L (1k-5k)	38.0	34.5	18.5	1,152	22.6%
XL (5k+)	32.5	31.8	24.1	428	68.2%

Table 10: Table 10 RQ3: match state against pull request size over the agent corpus, with the truncation rate per band. Truncation is caused by size and is therefore inseparable from the predictor, and the last column bounds the resulting artifact.

RQ	Finding	Effect
RQ1	Agent PRs match more often than matched human PRs on every pair, so the preregistered direction is reversed	+16.3 to +18.0pp
RQ1	Standardizing on observable artifacts (issue link, tests, size) reduces it by 34% to 60%	+6.8 to +10.8pp (40–66% retained)
RQ1	The pair decided by rule rather than by the model shows no difference that excludes zero once standardized	$-$ 1.2pp [ $-$ 3.9, +2.4]
RQ1	Paired overall score, agent $-$ human	+0.91 [+0.74, +1.08]
RQ2	Requirement $\leftrightarrow$ Acceptance is the pair agents most often leave build-only, in 100% of resamples	71.6% build-only
RQ2	The middle two pairs are not separable, so the locus claim holds for the end points only	78.9% full ordering stability

Table 11: Table 11 Summary of findings . Each finding is an observation made with an instrument that did not meet its calibration criteria (Section 5.1)

Figure 9: The three readings of the RQ1 difference on one scale. Hollow points are crude differences and filled points are differences after direct standardization for issue linking, test shipping and size band (Section 5.2). The dashed grey row is the rule-decided pair. Dotted lines mark the instrument’s cohort drift in the calibration (Section 5.1.2) against each rater’s labels and the adjudicated labels. The drift is pooled over all pairs on 25 pull requests and is shown as a descriptive reference only, not as a per-pair correction or a statistical bound

Automated Code Review

Automated code review uses software analysis and AI to find defects, assess changes, and support pull-request decisions.

See all 50 papers

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse
Jul 8, 2026Shyam Agarwal, Courtney Miller, Christian Kästner +1Software Engineering Coding Agents
Carnegie Mellon University
When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests
Feb 23, 2026Costain Nachuma, Minhaz ZibranAI Coding Agents AI-Assisted Software Development
Idaho State University, USA
How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses
Feb 19, 2026Kan Watanabe, Rikuto Tsuchida, Takahiro Monno +5AI Coding Agents Human-AI Collaboration
Nara Institute of Science and Technology, Japan

Jul 8, 2026J/K move · Enter open · S save

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Shyam Agarwal, Courtney Miller, Christian Kästner, Bogdan Vasilescu

Carnegie Mellon University

Software Engineering Coding Agents Human-AI Collaboration

Open paper AskPDF arXiv

Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built. Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. A motivating observational analysis of public GITHUB activity finds that agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones, yet the direction of these trends flips under different but equally defensible analysis choices, so the traces establish what is changing without explaining why. To recover the mechanisms, we synthesize practitioner discourse at scale into an explanatory theory: we collect 38,709 grey-literature documents (engineering blogs and Reddit threads), filter to those substantively about code review, and code a stratified random sample of 3,100 with an LLM-assisted pipeline, from which we build a causal model of 26 constructs and 67 relationships (64 directed, 3 contested). Its organizing claim is that review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. The theory makes the competing positions explicit and turns "AI is changing code review" into falsifiable propositions with named constructs and moderators. As a secondary contribution, we offer the underlying LLM-assisted, grey-literature theory-building method as a scalable template for software-engineering research, with a public implementation.

Abstract

Automated Code Review

Explore similar work

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests

How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses