Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

Xin Sun; Di Wu; Sijing Qin; Isao Echizen; Abdallah El Ali; Saku Sugawara

cs.AIApr 7, 2026

Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

Authors: Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah El Ali, Saku Sugawara

Organizations: National Institute of Informatics (NII), Japan · University of Amsterdam, the Netherlands · Hitotsubashi University, Japan · University of Tokyo, Japan · Centrum Wiskunde & Informatica (CWI), the Netherlands · Utrecht University, the Netherlands

Abstract

Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge). This work challenges its reliability by showing that trust judgments by LLMs are biased by disclosed source labels. Using a counterfactual design, we find that both humans and LLM judges assign higher trust to information labeled as human-authored than to the same content labeled as AI-generated. Eye-tracking data reveal that humans rely heavily on source labels as heuristic cues for judgments. We analyze LLM internal states during judgment. Across label conditions, models allocate denser attention to the label region than the content region, and this label dominance is stronger under Human labels than AI labels, consistent with the human gaze patterns. Besides, decision uncertainty measured by logits is higher under AI labels than Human labels. These results indicate that the source label is a salient heuristic cue for both humans and LLMs. It raises validity concerns for label-sensitive LLM-as-a-Judge evaluation, and we cautiously raise that aligning models with human preferences may propagate human heuristic reliance into models, motivating debiased evaluation and alignment.

Figures & tables

Figure 1: Three studies examine how source labels affect trust judgments by an LLM-as-a-judge and investigate the alignment with human behavior via mechanistic analyses of gaze attention, model self-attention, and logits entropy.

Figure 2: An example heatmap of gaze points on stimuli that are displayed on the lab monitor during the human assessment. Three AoIs are predefined: AoI-Label is the area for presenting disclosed label; AoI-Content is the area for presenting health information; AoI-Rating is the area to rate trust level in information on the left side.

Dependent Var.	Independent Var.	Condition	Mean (SD)
Trust score	Label (regardless of source)	Human	3.80 (.61)
		AI	3.67 (.65)
		Pairwise comparison: $p$ -value / effect (Std. $\beta$ )	.01 / .23 (medium)
	Source (regardless of label)	Human	3.62 (.64)
		LLM	3.85 (.60)
		Pairwise comparison: $p$ -value / effect (Std. $\beta$ )	<.001 / .35 (medium)

Table 1: Human trust scores by the disclosed label and actual source with pairwise

2\times 2

comparisons. Trust is higher under Human labels than AI labels, while higher for LLM-sourced answers than human-sourced answers.

Figure 3: LLM-as-a-Judge trust scores. (Left) Main effects of disclosed label (Human vs. AI) and (Right) content source (Human vs. LLM) on trust ratings by Wilcoxon signed rank test Rosner et al. (2006) with corrections. Horizontal brackets indicate statistically significant pairwise differences (*** $p<0.001$ , * $p<0.05$ ).

Figure 4: (Left): LLM’s attention allocation between two AoAs (i.e., label vs. content) and (Right): LLM’s logit entropy, across two label conditions (i.e., Human vs. AI). (** $p$ <.01, * $p$ <.05).

Figure 5: Analyses by GEE test Hardin and Hilbe (2012) with FDR correction Haynes (2013) of human gaze patterns (i.e., fixation count and fixation duration) in two AoIs. (** $p$ <.01, * $p$ <.05, “ns” is not significant).

Figure 6: LLM attention distribution density across AoAs (Label AoA vs. Content AoA) and two label conditions (Human vs. AI) across LLMs.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Demographic	Category	n (%)
Gender ( N =40)
	Female	23 (57.5)
	Male	16 (40.0)
	Non-binary	1 (2.5)
Age
	18–24	23 (57.5)

Appendix

Table 2: Participant characteristics in Study 1 (Sec. 3.1 ).

Figure 7: Instruction shown to the participants during the lab session before the human study.

Figure 8: Human trust ratings in Study 1. Top: Trust score distributions for the four ( $2\times 2$ ) conditions crossing true answer source (Human vs. LLM) and disclosed label (Human vs. AI). Bottom: Main effects collapsed across the other factor: trust scores by true source (regardless of label) and by disclosed label (regardless of true source). Violins show density; red dots indicate means; black lines indicate medians; thick bars denote interquartile ranges (IQR). Horizontal brackets mark significant pairwise differences ( ${}^{**}p<.01$ , ${}^{*}p<.05$ ).

Field	Template
Prefix	You are an evaluator of health information. Given a health question and its answer, you should provide your perceived trust ratings.
Question $Q$	{QUESTION} , e.g., “What is (are) Low Blood Pressure?”
Label $L$	{LABEL_TEXT} , e.g., “Source of the following answer: Artificial Intelligence-Generated”
Answer $A$	{ANSWER} , e.g., “You’ve probably heard that high blood pressure is a problem. Sometimes, blood pressure that is too low can also cause problems. Blood pressure is the force of your blood pushing against the walls of your arteries. Each time your heart beats, it pumps out blood into the arteries. ……”
Task	Rate the information rigorously and strictly based on the following assessment items using a 1-5 Likert scale: 1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree. - Fact: The information contains facts rather than opinions - Impartiality: The information is impartial - Credibility: I feel I can believe the information - Quality: The information has good quality - Objectivity: The information is objective - Comprehensiveness: The information is comprehensive - Reliability: The information is reliable - Trustworthiness: I trust this information - Usefulness2: The information tells me most of what I need to know - Usefulness1: The information helps me to understand the issue better - Clarity: The structure of the information is clear - Understandability: I can understand the information easily - Readability: I can read the information easily Return ONLY a JSON object with integer values between 1 and 5 (no extra keys).

Appendix

Table 3: LLM-as-a-Judge prompt template used in Study 2 (Sec. 3.2 ) and Study 3 (Sec. 3.3 ).

Figure 9: Trust scores of LLM-as-a-Judge under three label conditions from Study 3 (Sec. 3.3 ). Mean trust ratings (with error bars) produced by each model when the same health QA content is shown across three labels: Human , AI , and a non-semantical Placebo label as “[TAG]”. Horizontal brackets indicate significant pairwise differences ( ${}^{***}p<.001$ , ${}^{**}p<.01$ , ${}^{*}p<.05$ ).

Figure 10: LLM’s attention allocation between two AoAs (i.e., Label AoA vs. Content AoA) across three label conditions (i.e., Human vs. AI vs. Placebo). (*** $p$ <.001, ** $p$ <.01, * $p$ <.05)

Figure 11: LLM’s logit entropy between two AoAs (i.e., label vs. content) across three label conditions (i.e., Human vs. AI vs. Placebo). (*** $p$ <.001, ** $p$ <.01)

Figure 12: LLM attention distribution density across AOAs (Label AoA vs. Content AoA) and three labels (i.e., Human vs. AI vs. Placebo) across LLMs.

Explore similar work

May 28, 2026

Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs

As AI-generated and AI-assisted content floods online spaces, source labels attached to such content can distort human reasoning judgments, with downstream consequences for moderation, evaluation, and decision-making. Whether LLMs share this vulnerability, or offer more source-agnostic evaluation, remains an open question with direct implications for human-AI collaboration. We examine this issue using logical fallacies as a controlled setting to isolate source-label effects on reasoning quality, independent of domain knowledge. We conduct an online study (N=505) where participants are assigned to a source condition (human, AI, human with AI assistance, AI with human assistance, or no disclosure) and evaluate comments containing logical fallacies, comparing their judgments with those of LLMs (GPT-5.2, Gemini 2.5 Flash, Claude Sonnet 4.5), who were evaluated across the same source conditions. Human evaluators were significantly more susceptible to fallacies labeled as written by human or human with AI assistance and assigned higher trust and evaluation ratings in these conditions. LLM evaluations remained comparatively stable across source labels, though performance varied across models. Confidence levels were similarly high across conditions for both humans and LLMs, regardless of fallacy presence. Our findings indicate that source-label bias in reasoning evaluation is primarily a human vulnerability and highlight the potential of human-LLM collaboration in increasingly AI-mediated environments.

Mahjabin Nahar, Nafis Irtiza Tripto, Aiping Xiong +2

The Pennsylvania State University

LLM Evaluation Language Model Bias Evaluation

Ask PDF

Aug 21, 2026

When Trust Meets Truth: Trust-Truth Separability in LLM-as-Judge

LLM-as-Judge systems can produce multi-dimensional evaluations, such as trustworthiness, reliability, and factuality, and these outputs are often interpreted as independent evidence. We test this assumption for a common pair of judgments: trust scoring and binary truth classification. On correctness-controlled QA, LLM judges align trust scores with truth verdicts more tightly than human behavioral reference, suggesting weaker separations between trust and truth judgment. We then apply stress tests by changing only source cues of identical QA between Human and AI. Source attribution shifts not only trust scores but also truth verdicts and logit-derived correct-side probabilities. Results show that current LLM-as-Judge protocols should not treat trust scores as independent evidence for truth judgments.

Xin Sun, Di Wu, Yuchen Guo +4

National Institute of Informatics (NII), Japan · University of Amsterdam, the Netherlands · University of Tokyo, Japan +3

LLM-as-a-Judge Trust in AI Bias in LLM-as-a-Judge

Ask PDF

Jun 18, 2026

AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

Large language models (LLMs) are increasingly used as judges for open-ended generation, as large-scale human evaluation is often expensive and difficult to scale, yet their preferences remain imperfect proxies for human judgment. Existing auditing pipelines often assume that a reliable subset of examples or clean supervision signals are available beforehand, for example from human annotation, heuristic filtering, or the outputs of strong judges. In LLM evaluation, this assumption is fragile: the initial split may inherit judge bias, while human verification is typically too scarce to define stable groups at scale. We propose AURA, an adaptive uncertainty--aware refinement framework for auditing pairwise LLM--as--a--judge decisions under selected human verification. AURA iteratively learns a human-consistency signal, propagates reliable evidence, and prioritizes uncertain comparisons for human review. The key idea is to treat trust in a judge as a latent quantity that is progressively refined as evidence accumulates. We provide a compact formulation, a stable refinement procedure, and a comprehensive evaluation on both synthetic and real pairwise LLM-answer data.

Zilong Zhang, Yi-Ting Hung, Weiyi He +3

Department of Mathematics and Statistics Georgia State University · Department of Probability and Statistics Department of Computer Science and Engineering Michigan State University · Department of Mathematics and Statistics Concordia University +1

LLM-as-a-Judge LLM Auditing Active Learning Human-in-the-Loop Evaluation LLM Uncertainty Estimation

Ask PDF