The Quality of Claude AI-authored Python Tests Is Not Weaker Than Human-authored Tests

Douglas J. Leith

cs.SEAug 15, 2026

The Quality of Claude AI-authored Python Tests Is Not Weaker Than Human-authored Tests

Authors: Douglas J. Leith

Organizations: Trinity College Dublin Dublin, Ireland

Abstract

We evaluate the quality of Claude AI-written Python tests against human-written Python tests from two established open-source projects Django and Pandas. Hundreds of tests per corpus are scored under one identical protocol. Using one-sided non-inferiority bounds, we find that the tests written by recent Claude models (Sonnet/Opus 4.6 and later) are no weaker than the two human-written corpora. In this study: (i) the AI-written corpus is tests from real tools, not synthetic tests generated in isolation against a fixed target, the setup used by every other AI-test-generation study we are aware of; (ii) every test is individually scored under three independent fault-injection protocols plus a seven-axis qualitative design rubric, allowing methods to cross-validate each other; (iii) tests are scored individually, rather than suite-level, identifying exactly which specific tests need attention.

Figures & tables

Corpus	Written by	Domain	Size (src/test LOC; tests)
Extractor	Claude (Sonnet 4.6)	ARM64/ObjC reverse eng.	21k/20.7k; 1,423 tests
Django	Human; ~88k GitHub stars; active since 2012	Web framework	165k/354k; 17,519 tests
Pandas	Human; ~49k GitHub stars; active since 2010	Data analysis library	263k/407k; 20,244 tests

Table 1. Corpus scale: authorship, domain, and code/test size 2 2 2 Note that these LOC numbers include comments/docstrings and white space. for the three corpora studied.

Metric	Extractor	Django	Pandas
Scored	298	350	350
EFFECTIVE	288	342	334
SURVIVED	9	7	11
UNREACHED	1	1	5
EFFECTIVE Rate	96.6%	97.7%	95.4%
95% CI (Wilson)	93.9–98.2%	95.6–98.8%	92.7–97.2%

Table 2. Historical-revert testing results by corpus.

Metric	Extractor	Django	Pandas
Per-commit
Commits	48	100	116
Tests	234	100	116
Scored mutants	695	273	291
Killed	370	131	131
Survived	325	142	160

Table 3. AST mutation testing results by corpus, per-commit and coverage-based protocols 9 9 footnotemark: 9 , 10 10 10 Observe that for Extractor the number of tests is larger than the number of commits. Django and Pandas had one test per commit, whereas Extractor often included multiple tests in one commit. .

Metric	Extractor	Django
Scored	679	679
Killed	399	396
Survived	280	283
Kill rate	58.8%	58.3%
95% CI (Wilson)	55.0–62.4%	54.6–62.0%

Table 4. Whole-test-suite mutation kill rate, Extractor vs. Django utils package.

Axis	Verdict scale	What it asks
Intent fidelity	MATCH / PARTIAL / MISMATCH	Does the assertion actually check what the docstring/name/comment claims?
Assertion strength	STRICT / MODERATE / WEAK / n/a	Is the assertion at the ceiling of precision the claim calls for, without introducing fragility it does not need?
Behaviour/implementation coupling	BEHAVIOR / JUSTIFIED-MECHANISM / UNJUSTIFIED-COUPLING	Does it assert an externally observable outcome, or an incidental implementation detail that would break under a legitimate refactor?
Anti-pattern	present / absent per category	A checklist for named smells: dead lever, assertion masking (coverage-real or diagnostic-only), over-broad mock, swallowed exception, log-instead-of-assert, tautology, skip/xfail masking
Setup/fixture proportionality	OK / DEFECTIVE	Does Arrange build only what is needed, and does it construct the scenario the test actually claims to exercise?
Purpose stated	CLEAR (per-test/class/module/external) / PARTIAL / WRONG / NONE	Does some source (docstring, comment, commit message, or a linked issue tracker) state what the test verifies and why?

Table 5. The seven-axis qualitative rubric: what each axis asks, and its verdict scale.

	Intent-fidelity	Assertion strength	Coupling	Anti-pattern	Setup	Purpose-stated	Cohesion
Extractor (n=298, critical=6 (2.0%), moderate=58 (19.4%))
Critical	4	6	0	1	2	0	0
Moderate	9	37	0	18	1	1	1
Minor	0	0	0	5	0	3	0
Info	0	0	11	0	0	5	0
Total	13	43	11	24	3	9	1

Table 6. Per-axis severity breakdown, by corpus 11 11 11 Note that a test is counted under every axis where it scored non-ideal, so two independently real properties of the same test, e.g. a weak assertion and a thin docstring, are counted separately in the row tallies. The headline numbers, e.g. critical=6 for Extractor, report the actual number of tests affected. .

Corpus	Flagged by both	Qualitative only	Historic-revert only
Extractor	2/2 (100%)	5/5 (100%)	2/6 (33.3%)
Django	3/3 (100%)	6/6 (100%)	2/12 (16.7%)
Pandas	0/0 (–)	3/3 (100%)	7/15 (46.7%)

Table 7. Tests confirmed genuinely defective / tests flagged, by which methodology flagged them, by corpus. “Qualitative” = assertion-strength WEAK; “Historic-revert” = SURVIVED/UNREACHED (non-EFFECTIVE).

Corpus	Qualitative FP rate	Historical-revert FP rate
Extractor	0/7 (0%)	4/8 (50.0%)
Django	0/9 (0%)	10/15 (66.7%)
Pandas	0/3 (0%)	8/15 (53.3%)

Table 8. False-positive rate of each triage methodology, by corpus.

Panel of Figure 1 — Figure 1. Kill-rate distributions for Extractor, with defective tests marked by dashed line.

Corpus	Per-commit	Coverage-guided	Combined unique
Extractor (AI)	3/25	3/25	5/42
Pandas (human)	4/25	2/25	5/46
Django (human)	0/25	1/25	1/50

Table 9. Low-kill-rate triage defect-find rate, by corpus.

Corpus	Written by	Domain	Size (src/test/AI-test LOC; tests/AI-tests
Quay (quay/quay)	Claude Opus 4.6, Sonnet 4.6	Cloud infrastructure	$\sim$ 128,300/ $\sim$ 79,700/ $\sim$ 3,600, $\sim$ 3,800/214
AWX (ansible/awx)	Claude Opus 4.6/4.8, Fable 5	IT automation	$\sim$ 104,900/ $\sim$ 58,400/ $\sim$ 1,200, 2,674/102

Table 10. Details of Quay and AWX.

Figure 2. Tests introduced per individual commit, sorted in descending order, for Quay, AWX, and Extractor.

Corpus	Tests (SCORED)	Scored mutants	Killed	Survived	Kill rate	95% CI (Wilson)
Quay	128	1,825	817	1,008	44.8%	[42.5%, 47.1%]
AWX	50	665	391	274	58.8%	[55.0%, 62.5%]

Table 11. AST mutation testing results by corpus, coverage-based protocol.

Corpus	vs. Django	vs. Pandas
Quay	7.8%	3.8%
AWX	0%	0%
Extractor	5.9%	2.0%

Table 12. One-sided 95% non-inferiority bounds on the coverage-guided kill-rate, by corpus.

	Intent-fidelity	Assertion strength	Coupling	Anti-pattern	Setup	Purpose-stated	Cohesion
Quay (n=214, critical=8 (3.7%), moderate=39 (18.2%))
Critical	7	4	0	2	5	0	0
Moderate	17	27	2	5	4	4	1
Minor	0	0	0	3	1	3	0
Info	0	0	13	0	0	45	0
Critical+Moderate	24	31	2	7	9	4	1

Table 13. Per-axis severity breakdown, by corpus.

Corpus	vs. Django	vs. Pandas
Quay	4.8%	5.6%
AWX	4.6%	5.6%
Extractor	2.3%	3.2%

Table 14. One-sided 95% non-inferiority bounds on the qualitative-rubric critical-defect rate, by corpus. Tests are attributable to Opus 4.6 and later.

	Intent-fidelity	Assertion strength	Coupling	Anti-pattern	Setup	Purpose-stated	Cohesion
Quay (n=233, critical=24 (10.3%), moderate=67 (28.8%))
Critical	17	14	0	11	13	3	0
Moderate	22	52	5	8	7	7	3
Minor	0	2	0	1	4	4	0
Info	0	0	9	0	0	18	0
Critical+Moderate	39	66	5	19	20	10	3

Table 15. Per-axis severity breakdown for Quay tests attributable to Opus 4.5 and earlier.

Severity	Definition
Critical	A genuine behavioural defect: axis-1 MISMATCH , axis-5 DEFECTIVE with no mitigation, or an axis-2 strength WEAK or MODERATE proven to have missed a real mutation.
Moderate	A real, presentable gap, with no proof yet of an actual missed defect.
Minor	Diagnostic-only assertion masking, a tautology, or an axis-6 WRONG / PARTIAL docstring that does not affect the test’s own correctness.
Informational	axis-3 JUSTIFIED-MECHANISM , axis-6 CLEAR_COMMIT_TEXT when it’s the
row’s only flagged reason; any finding whose own note calls the deviation borderline/trivial.

Table 16. Severity tier definitions for non-ideal rubric verdicts.

Severity	Total tests	Unchanged (A=B)	Changed ( $\geq 1$ axis downgraded)
Extractor
critical	6	4	2
moderate	58	52	6
Django
critical	5	3	2
moderate	44	37	7

Table 17. Changes in qualitative rubric scoring from Stage A to Stage B.

Severity	Axis changed	# tests	Direction
Extractor
critical	axis2 (assertion strength)	2	STRICT -> WEAK / MODERATE
moderate	axis2 (assertion strength)	5	STRICT -> MODERATE (4×) / WEAK (1×)
moderate	axis5 (scenario realism)	1	OK -> DEFECTIVE
Django
critical	axis5 (scenario realism)	2	OK -> DEFECTIVE

Table 18. Breakdown by rubric axis of changes in scoring from Stage A to Stage B.

Test (commit)	Historic-revert	Severity	Commentary
Weak-value/loose-bound oracle
test_flags_struct_behind_name_still_detected	S	moderate	There is one assertion, and the pre-commit (loose) and post-commit casing rules both return True for the input used by the test.
test_forward_branch_handler_dominance	E/New	critical	Both test assertions are structurally non-discriminating.
test_by_file_breaks_the_package_merge_cycle	E	moderate	Test assertions only check import-set membership on two files and never check for all 3 expected filenames. EFFECTIVE only because reverting the commit is a mutation that these assertions happen to catch.
test_qualified_keys_avoid_short_name_collision	E	moderate	Reverting the commit re-merges three types into one, which the test’s count check catches, hence EFFECTIVE. But the test never runs the other half of the fix, the code that fills in field data without guessing between same-named types; breaking that half goes unnoticed.
Wrong entry point / bypassed wrapper

Table 19. Extractor tests flagged assertion-strength WEAK by the qualitative rubric, with historical-revert outcomes. Historic-revert: S = SURVIVED, E = EFFECTIVE, E/New = EFFECTIVE (TESTING_NEW_FUNCTION).

Test (commit)	Historic-revert	Severity	Commentary
Weak-value/loose-bound oracle
test_flags_struct_behind_name_still_detected	S	moderate	See Table 19 .
test_varint_tag_shift_is_plain	S	–	Test claims to check one specific safeguard, but its input also differs from the bad case in a second, unrelated way. That second difference is enough to make the test pass, whether or not the safeguard works.
test_index_scale_lsl3_on_nonfield_reg_does_not_block_fixed	S	moderate	The test’s one assertion (a decoy index-scale lsl doesn’t block fixed32 classification) passes equally whether the terminator’s register-matching is a genuine, selective gate or simply absent.
test_lendelim_tag_wire_type_2_is_plain	S	moderate	See Table 19 .
No defect

Table 20. Extractor tests flagged non-EFFECTIVE by historical-revert, with qualitative rubric scores. Historic-revert: S = SURVIVED, U = UNREACHED.

Test (commit)	Historic-revert	Severity	Commentary
Weak-value/loose-bound oracle
test_change_password	E	critical	The test only checks assertIsInstance(user, User) after a password change, but it never actually changes the password, so the session hash it is meant to validate never diverges regardless of whether the fix’s await request.auser() is present or reverted to sync request.user . EFFECTIVE here is coincidental: reverting the whole commit instead removes the auser() call entirely, crashing with AttributeError before the weak assertion (or the narrow oracle gap) is ever reached.
test_aggregate_subquery_annotation	S	critical	assertLessEqual(sql.count('SELECT'), 4) accepts both the buggy count (4) and the fixed count (3), so it cannot catch the bug it was written for.
test_remove_unique_together_on_pk_field	S	critical	Fully reverting the commit’s fix, removing the exact behaviour the test is meant to guard, still leaves the test passing.
test_rename_model_with_db_table_rename_m2m	E	moderate	The test only checks that the rename completes without raising. A concrete, equally simple check is available and unused: asserting the actual row count on the renamed relation, which would additionally catch corruption that doesn’t raise.
test_namespaced_db_table_foreign_key_reference	E	moderate	The test has no explicit assertions, it passes as long as the model/field creation calls complete without raising.

Table 21. Django tests flagged assertion-strength WEAK by the qualitative rubric, with historical-revert outcomes. Historic-revert: S = SURVIVED, E = EFFECTIVE.

Test (commit)	Historic-revert	Severity	Commentary
Dead assertion order (stacked context managers)
test_json_response_raises_type_error_with_safe_arg	S	critical	The test wraps two checks in one with (assertRaisesMessage(...), assertWarnsMessage(...)): block. Python exits stacked context managers in reverse order, so the warning check (entered second) exits first, sees the exception still active, and skips its own check; the warning half can never fail. The other half also cannot distinguish old from new code: it passes safe=True explicitly, and both versions raise the same error for that regardless of the fix.
Weak-value/loose-bound oracle
test_parameter_count_exceeds_variable_or_column_limit	S	moderate	See Table 21 .
test_aggregate_subquery_annotation	S	critical	Same test as the WEAK-axis table above.
test_remove_unique_together_on_pk_field	S	critical	Same test as the WEAK-axis table above.

Table 22. Django tests flagged non-EFFECTIVE by historical-revert, with qualitative rubric scores. Historic-revert: S = SURVIVED, U = UNREACHABLE.

Test (commit)	Historic-revert	Severity	Commentary
No assertion beyond non-crash
test_pass_colormap_instance	E	moderate	Calls df.plot.scatter / hexbin with a Colormap instance (rather than a string name) and checks nothing beyond the call not raising. A concrete stricter check exists and is mechanically straightforward: ax.collections[-1].get_cmap() is cmap on the returned Axes .
test_interchange_from_corrected_buffer_dtypes	E	moderate	Monkeypatches a string column’s reported buffer dtype to simulate a corrected interchange scenario, then calls pd.api.interchange.from_dataframe(df) without even assigning the return value: the only signal is that the call doesn’t raise.
test_xz_compression_level_read	E	moderate	The size-comparison assertion ( compressed_size_default < compressed_size_fast ) is well-calibrated, but the to_csv branch’s read-back call, pd.read_csv(path, compression="xz") , is never checked against the original data: unlike the file’s sibling round-trip test, which does assert equality.

Table 23. Pandas tests flagged assertion-strength WEAK by the qualitative rubric, with historical-revert outcome. Historic-revert: E = EFFECTIVE.

Test (commit)	Historic-revert	Severity	Commentary
Wrong entry point / bypassed wrapper
test_fillna_dict_inplace_nonunique_columns	S	moderate	The test’s three duplicate "A" columns sit adjacent, so df.columns.get_loc("A") returns a plain contiguous slice. For a contiguous slice, the old single-write and the new per-location-loop write touch the same positions in the same order and produce identical output. The fix’s real target, non-contiguous duplicate columns, is never constructed here.
test_astype_numpy_to_ea	U	informational	For this test’s direction (numpy int64 -> Int64Dtype ), the code finds a value before the buggy fallback line runs, so that line is never reached. The typo this commit fixes only manifests in the reverse direction, which this test does not exercise.
test_dt_round_nonnano_higher_resolution_no_op	S	informational	The fix is a one-line change inside the array-level _round() , and the test’s assert not np.shares_memory(...) looks like a precise oracle for exactly that line, but it observes memory identity only after the result passes back up through the .dt accessor’s Series-construction wrapper, one layer above the fixed line, which already produces a fresh array regardless of what the inner method returns.
test_duration_fillna_numpy	S	–	The test fills with a whole TimedeltaArray , not a scalar Timedelta . The commit’s refactor only changes behaviour for a scalar- Timedelta -with-unit input; the array-shaped branch this test exercises is identical in both old and new code.
test_astype_dt64_to_int64	S	–	M8[ns] -> int64 is a lossless direct cast under both the removed and the new code path for this test’s dtype pair, including the non-NaT-fill-value case this commit adds. The original bug was that old code raised for this dtype combination, not that it computed a wrong value; this test’s pairing does not trigger the raise either way.

Table 24. Pandas tests flagged non-EFFECTIVE by historical-revert, with qualitative rubric scores. Historic-revert: S = SURVIVED, U = UNREACHED.

Test (commit)	Historic-revert	Kill rate	A2 Strength	Category	Commentary
test_varint_tag_shift_is_plain †	S	0/14	STRICT	Weak-value/loose-bound oracle	The test’s input differs from the bad case in a second, unrelated way that alone makes it pass, masking whether the actual safeguard works.
test_no_sibling_data_ivar_is_not_phantom	E/New	0/4	STRICT	Weak-value/loose-bound oracle	The fixture’s dict lookup misses regardless of how the surrounding guards are perturbed, so the negative assertion cannot discriminate correct from broken.
test_real_vmaddr_is_not_a_bind	E/New	0/11	STRICT	Weak-value/loose-bound oracle	The fixture value sits far from the 1<<62 boundary the guard exists to test, so the guard’s own comparison is never exercised at a discriminating point.
test_resolve_image_exact_full_path †	E/New	0/8	MODERATE	Wrong entry point / bypassed wrapper	The single-image fixture only ever exercises the exact-match tier; it cannot tell whether the other two resolution tiers work at all.
test_oracle_contradiction_rejected	E/New	0/21	STRICT	Wrong entry point / bypassed wrapper	The code recovers a message’s field numbers, then double-checks them against a second, independent count of the same numbers, rejecting if they do not line up. The test’s example has the two counts share nothing in common at all, such an extreme mismatch that almost any check would catch it, so passing does not prove the actual comparison is correct.

Table 25. Extractor tests found defective by per-commit low-kill-rate triage. Historic-revert: S = SURVIVED, E/New = EFFECTIVE (TESTING_NEW_FUNCTION).

Test (commit)	Historic-revert	Kill rate	A2 Strength	Category	Commentary
test_varint_tag_shift_is_plain †	S	0/38	STRICT	Weak-value/loose-bound oracle	Same defect as in table above
test_no_sibling_data_ivar_is_not_phantom	E/New	0/4	STRICT	Weak-value/loose-bound oracle	Same defect as in table above.
test_real_vmaddr_is_not_a_bind	E/New	0/3	STRICT	Weak-value/loose-bound oracle	Same defect as in table above.
test_resolve_image_exact_full_path †	E/New	12/40	MODERATE	Wrong entry point / bypassed wrapper	Same defect as in table above.

Table 26. Extractor tests found defective by coverage-guided low-kill-rate triage. Historic-revert: S = SURVIVED, E/New = EFFECTIVE (TESTING_NEW_FUNCTION).

Pattern	n	%
BUG_FIX_REGRESSION	45	37%
OTHER	33	27%
FEATURE_IMPLEMENTATION_UNPROMPTED	21	17%
EXPLICIT_TEST_REQUEST	16	13%
COVERAGE_REVIEW_REQUEST	6	5%

Table 27. How Extractor test-writing episodes were prompted, by pattern.

Software Engineering

Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.

See all 274 papers

Sep 11, 2026

What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whether their code differs from human code in the quality dimensions dominating lifecycle cost. We compare human-written and AI-generated code at scale: 787,562 function pairs across Python, Java, and C, each human function mined from open-source repositories paired with implementations generated from its docstring by three AI assistants (OpenAI GPT models, DeepSeek-Coder, Qwen2.5-Coder). We characterize structural complexity and statistical naturalness, and map static-analysis findings onto Orthogonal Defect Classification for defects and the Common Weakness Enumeration for vulnerabilities, making authors and languages directly comparable. AI-generated code is structurally compressed and stylistically templated: roughly half the size and branching of human code, clustering apart at the style level. Defect profiles differ in kind: human code concentrates issues of mature codebases, AI code repetitive boilerplate; security is language-dependent, with LLMs producing more, and more severe, findings in Python and Java but fewer high-severity memory-safety findings than humans in C. Once size is controlled for, complexity metrics carry little signal, while naturalness separates authors. Finally, we release CQBench, a benchmark of 27,346 issue-prone tasks with baselines and an evaluation pipeline for quality assurance and security testing.

Cristina Improta, Pietro Liguori, Domenico Cotroneo

University of Naples Federico II, Naples, Italy · University of North Carolina at Charlotte, Charlotte, NC, USA

LLM Security Code Generation Evaluation Code Generation Benchmarks

Ask PDF

Apr 20, 2026

Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation

AI coding assistants increasingly generate code alongside tests. How developers structure test code, whether inline with the implementation or in separate blocks, has traditionally been a matter of testing philosophy. We investigate whether this choice affects AI code generation quality. We conduct a large-scale empirical study (830+ generated files, 12 models, 3 providers) using SEGA, a three-dimensional evaluation framework measuring Determinism, Preservation, and Correctness. Comparing inline test syntax (Python doctests) against separated test syntax (Rust #[test] blocks) on a d-ary heap implementation, we find that: (1) inline tests yield near-perfect preservation (100%) and correctness (92-100%) across all models; (2) separated tests expose stark model-tier gaps (0-100% correctness) and independence between preservation and correctness; (3) model behavior evolves across generations, and notably one model breaks the test suppression pattern of its three predecessors; (4) mechanistic analysis on 7 open-source architectures (6 transformers and a gated-linear Recurrent Neural Network (RNN)) reveals inline test markers receive 2.8-4.4

\times

stronger attention in 5/7 models, with causal validation via knockout and steering experiments on the 4 code-specialized transformers and RWKV-6; the co-location mechanism extends to a non-transformer architecture, suggesting the design recommendation is robust to future architectural shifts. In the Foundation Model era, test syntax structure is a software design concern: co-locating tests with implementation code produces measurably better AI-generated code. This arxiv long version includes appendices that further qualify the effect as bounded by both model capability and programming language.

Éric Jacopin

Cosmic AI

Code Generation Code Generation Evaluation Code Language Models

Ask PDF

Sep 24, 2026

Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.

Douglas Leith

Trinity College Dublin Dublin, Ireland

AI Agent Reliability Data Provenance AI Coding Agents LLM Reliability Code Generation Evaluation AI-Assisted Software Development

Ask PDF

Abstract

Software Engineering

Explore similar work

What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation

Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase