Measuring LLM Trust Allocation Across Conflicting Software Artifacts

Noshin Ulfat; Ahsanul Ameen Sabit; Soneya Binta Hossain

cs.SEApr 3, 2026

Measuring LLM Trust Allocation Across Conflicting Software Artifacts

Authors: Noshin Ulfat, Ahsanul Ameen Sabit, Soneya Binta Hossain

Organizations: University of Texas at Dallas Texas, USA · University of Texas at Dallas Richardson, Texas, USA

Abstract

LLM-based software engineering assistants often reason over multiple artifacts, including code, documentation, signatures, and tests, even when those artifacts are incomplete or mutually inconsistent. Existing evaluations primarily measure final outputs, leaving unclear whether a model recognized unreliable evidence, identified the faulty source, or prioritized the appropriate artifact. We introduce TRACE, a controlled method for evaluating how LLMs assess and prioritize conflicting software artifacts. TRACE constructs paired clean and perturbed versions of real-world Java method bundles by injecting known faults into the documentation, implementation, or both while holding the remaining artifacts fixed. Models then assess artifact quality, detect and localize inconsistencies, and rank the available sources by reliability. Using 22,339 valid responses from seven LLMs on 456 method bundles, we find that quality penalties are generally localized to the perturbed artifact and increase with fault severity. However, models exhibit a consistent source-origin asymmetry: they detect documentation faults at 67-94% and explicit documentation-implementation contradictions at 50-91%, but detection falls by 21-43 percentage points when only the implementation changes while documentation remains intact. Models also struggle to deprioritize faulty implementations, and confidence provides little separation between correct and incorrect judgments for six of seven models. These results show that current LLMs are not symmetric integrators of software evidence: they audit natural-language specifications more reliably than subtle implementation behavior. TRACE provides a controlled method for exposing this failure mode before LLMs are used in correctness-critical software engineering workflows.

Figures & tables

Variant	Operation	Severity
NoDesc	Remove Javadoc prose; retain tags	—
NoReturn	Remove @return , when present	—
NoDescReturn	Remove prose and @return	—
DocBug	Perturb Javadoc; retain the base MUT	H/N/S
MutBug	Perturb MUT; retain the base Javadoc	H/N/S
Conflict	Create a Javadoc–MUT conflict by changing Javadoc, MUT, or both	H/N/S

TABLE I: TRACE perturbation variants.

Fig. 2: Mean artifact-quality scores by model and dataset variant for Javadoc, signature, MUT, test prefix, and overall bundle. The figure shows model-specific score calibration and the artifacts receiving the largest quality penalties under each perturbation.

Fig. 3: Quality-score change relative to the base variant, $\Delta=\bar{a}_{m,v,x}-\bar{a}_{m,0,x}$ , by model, perturbation, and artifact dimension. Negative values indicate lower perceived quality; concentrated negative values indicate localized sensitivity to the perturbed artifact.

Fig. 4: Mean overall quality score by model and severity tier for DocBug , MutBug , and Conflict ; error bars show

\pm

1 standard deviation. Wider heavy-to-subtle gaps indicate stronger severity sensitivity.

Fig. 5: Description fidelity for Conflict traces, measured as cosine similarity to ground-truth fault descriptions ( $N=3{,}189$ ; embedding: BAAI/bge-base-en-v1.5 ). Panel (a) compares IR-strict detected and missed traces; detected traces have higher similarity on average ( $0.83$ , $n=2{,}477$ ) than missed traces ( $0.73$ , $n=712$ ). Panel (b) shows per-sample cosine distributions for PCA, IC, IR, and their combined representation.

Fig. 6: Confidence calibration on Conflict under IR-strict evaluation. Panel (a) compares mean confidence for detected and missed traces. Panel (b) shows the detected–missed confidence gap, with the number of missed samples shown below each bar. DeepSeek-V3.2-Speciale shows the largest separation (

0.73

); the other six models show little separation.

Fig. 7: Javadoc-versus-MUT detection asymmetry under IR-strict evaluation. Bars report net detection gain above the clean false-positive baseline. Javadoc faults are detected more reliably than MUT faults across all seven models: $+54.8$ – $+76.5$ pp for DocBug versus $+14.0$ – $+42.5$ pp for MutBug .

Fig. 8: Javadoc–MUT inconsistency detection by perturbation type, model, severity tier, and detection signal. Dashed lines show each model’s clean-sample false-positive rate; the main comparison is net detection gain above this baseline.

Fig. 9: Severity-stratified Javadoc–MUT contradiction detection under IR-strict evaluation (

n=152

per severity tier). Sonnet is most robust (

96\%\rightarrow 91\%\rightarrow 86\%

), followed by DS-V3.2 (

92\%\rightarrow 83\%\rightarrow 80\%

) and Haiku (

91\%\rightarrow 83\%\rightarrow 75\%

). GPT-4o shows the steepest drop (

76\%\rightarrow 41\%\rightarrow 32\%

). Opus is nonmonotonic (

77\%\rightarrow 81\%\rightarrow 76\%

) because many traces recognize the mismatch but fail the artifact-localization requirement.

Fig. 10: Description fidelity by severity for Conflict , measured as cosine similarity to ground-truth fault descriptions using BAAI/bge-base-en-v1.5 . Scores are shown for PCA, IC, IR, and their combined representation. Missing IC or IR descriptions receive zero similarity, so the metric reflects end-to-end trace fidelity.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 780 papers

Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware Engineering LLM-as-a-Judge
University of British Columbia, Canada
Comparing Developer and LLM Biases in Code Evaluation
Mar 25, 2026Aditya Mittal, Ryan Shar, Zichu Wu +6Bias in LLM-as-a-Judge Language Model Bias Evaluation
Carnegie Mellon University
Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence
Oct 23, 2026ZhiHong Liu, Shuyao GaoLLM Auditing Cognitive Bias in Language Models
Seoul School of Integrated Sciences and Technologies (aSSIST University), Seoul, Republic of Korea · Liaoning Mechanical & Electrical College of Technology, Dandong, Liaoning, China

Apr 18, 2026J/K move · Enter open · S save

Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

Zixiao Zhao, Amirreza Esmaeili, Fatemeh Fard

University of British Columbia, Canada

Software Engineering LLM-as-a-Judge Language Model Bias Evaluation Bias in LLM-as-a-Judge Code Generation Evaluation

Open paper AskPDF arXiv

Large Language Models are increasingly used as judges to evaluate code artifacts when exhaustive human review or executable test coverage is unavailable. LLM-judge is increasingly relevant in agentic software engineering workflows, where it can help rank candidate solutions and guide patch selection. While attractive for scale, current practice lacks a principled account of reliability and bias: repeated evaluations of the same case can disagree; small prompt edits can swing outcomes; and seemingly semantics-preserving, human-equivalent perturbations may elicit divergent verdicts. This paper studies LLM-as-a-Judge for code through a measurement-first lens. We analyze two pointwise judging regimes across code generation, code repair task, and test generation, and we systematically probe prompt-induced biases. Our study considers difficulty levels for repeated runs and controlled prompt interventions that isolate one presentation cue at a time, and it evaluates judges using consistency and sensitivity to bias. We find that judge decisions are highly sensitive to prompt biases even when the underlying code snippet is unchanged. Across all three tasks, several biases systematically shift preferences toward the option favored by the prompt, improving accuracy when that option aligns with the gold answer but substantially reducing it otherwise. In some settings, these effects are large enough to change task-level conclusions and alter relative model rankings. These findings show that reported judge performance may reflect prompt artifacts rather than stable assessment ability, posing a direct threat to the validity and reproducibility of code evaluation. We therefore argue that LLM-as-a-Judge studies should report bias sensitivity alongside accuracy and incorporate explicit controls to support more trustworthy model comparison in software engineering.

Abstract

LLM Reliability

Explore similar work

Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

Comparing Developer and LLM Biases in Code Evaluation

Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence