When Text and Numbers Disagree: Evidence Arbitration in Large Language Models

Mattia Carletti; Edward Phillips; Fredrik K. Gustafsson; Patitapaban Palo; Lei Clifton; Danielle Belgrave; Xiao Gu; David A. Clifton

cs.CLAug 20, 2026

When Text and Numbers Disagree: Evidence Arbitration in Large Language Models

Authors: Mattia Carletti, Edward Phillips, Fredrik K. Gustafsson, Patitapaban Palo, Lei Clifton, Danielle Belgrave, Xiao Gu, David A. Clifton

Organizations: Department of Engineering Science, University of Oxford, Oxford, UK · Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK · GlaxoSmithKline, London, UK · Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou, Jiangsu, China

Abstract

Large language models (LLMs) are increasingly used in settings where textual summaries, numerical observations, and external tool outputs may provide conflicting evidence. We study how LLMs arbitrate between such sources when they support opposing decisions. To do so, we introduce a controlled synthetic benchmark in which latent risk trajectories generate both numerical time series and natural language summaries, allowing us to construct conflicts where exactly one evidence source is aligned with the ground-truth label. This design lets us independently manipulate modality, temporal recency, source reliability, and evidence provenance. Across open-weight instruction-tuned models, we find that arbitration behaviour is systematic rather than random: models exhibit distinct text-versus-number preferences, follow temporal recency more consistently than explicit reliability cues, and can over-rely on external forecasts even when they conflict with direct contextual evidence. These results suggest that current LLMs often rely on heuristic arbitration strategies when integrating heterogeneous evidence, highlighting a failure mode for tool-augmented decision systems.

Figures & tables

Figure 1: Evidence arbitration under conflict. A numerical time series and a textual summary, drawn from different windows of a shared latent risk trajectory, support conflicting predictions at the target time

T{+}k

: high for the older numerical source, low for the newer textual source. The task requires the model to decide which source to prioritize when answering the binary-choice prompt.

Figure 2: Benchmark conflict settings for evidence arbitration. Each panel illustrates one of the four controlled conflict settings used in our evaluation. In every setting, two evidence sources support opposing decisions and exactly one source is aligned with the ground-truth label. (A) Baseline modality priors: both sources cover the same window

[0,T]

and merely disagree. (B) Temporal recency: sources cover different time windows; the more recent source is always aligned with truth. (C) Reliability: one source is marked as unreliable (a corruption note in text, missing values in numbers); the reliable source is aligned with truth. (D) Tool forecast: an external forecasting tool predicts a value at

T{+}k

that contradicts the observed context; context is aligned with truth.

Figure 3: Benchmark construction pipeline. A single example is traced through the three stages used to generate model prompts. (A) Time series generation: a sampled label, slope, and margin fix the target value

y_{T+k}=0.5\pm\text{margin}

, from which the observed trajectory is generated by integrating backwards with additive noise. (B) Text generation: trajectory-level features are discretized and rendered as a natural language summary using templates and synonym sampling. (C) Prompt generation: the textual summary and numerical series are combined with task framing, evidence-order controls, and answer choices to produce the final binary-choice prompt.

Panel of Figure 4 — Figure 4: Arbitration accuracy under conflicting evidence. Panels show classification accuracy across the four conflict settings and evidence-order conditions: (A–B) baseline modality priors, (C–D) temporal recency, (E–F) reliability, and (G–H) tool forecast conflicts. In each setting, exactly one evidence source is aligned with the ground-truth label (GT). Shaded bars show unimodal reference accuracy using only the ground-truth-aligned source, while solid bars show accuracy when both conflicting sources are provided. Results are averaged over three random seeds.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Representative examples of generated time series. Generated trajectories with observable length

T=16

and forecast horizon

k=1

Hyperparameter	Default Value
Answer choices order	“A” first
Answer choices-labels mapping	“A”= high
Sequence length	16
Forecast horizon	$k=1$
Frequency	1 minute
Domain	generic

Appendix

Table A1: Default hyperparameter configuration used in the main experiments unless otherwise specified.

Model	$\|\Delta\mathrm{Acc}\|$ numeric-only	$\|\Delta\mathrm{Acc}\|$ text-only	$\|\Delta\mathrm{Acc}\|$ conflict (numeric GT)	$\|\Delta\mathrm{Acc}\|$ conflict (text GT)
Domain Sensitivity
Qwen3-1.7B	$0.06\pm 0.01$	$0.01\pm 0.01$	$0.13\pm 0.07$	$0.14\pm 0.06$
Qwen3-4B	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.05\pm 0.04$	$0.05\pm 0.04$
Qwen3-8B	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.05\pm 0.07$	$0.05\pm 0.07$
Qwen3-14B	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.03\pm 0.03$	$0.03\pm 0.03$
Gemma-2-9B-It	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.04\pm 0.03$	$0.04\pm 0.02$

Appendix

Table A2: Absolute accuracy change (

|\Delta\mathrm{Acc}|

) under different sensitivity-analysis settings. Accuracy deltas are aggregated across sweep values, seeds, and (where applicable) evidence-order settings, and reported as mean

\pm

standard deviation. Lower values indicate lower sensitivity to the swept parameter. GT: ground truth.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
May 13, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleLLM-as-a-Judge Faithfulness of Language Model Explanations
Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.
Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
Feb 16, 2026Luise Ge, Yongyan Zhang, Yevgeniy VorobeychikLLM Decision-Making Framing Effects in Language Models
Washington University in St. Louis
Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts
Feb 25, 2026Jessica Y. Bo, Lillio Mok, Ashton AndersonHuman-AI Decision Making LLM Decision-Making
Computer Science University of Toronto

May 13, 2026J/K move · Enter open · S save

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

Riya Tapwal, Abhishek Kumar, Carsten Maple

Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.

LLM-as-a-Judge Faithfulness of Language Model Explanations Language Model Bias Evaluation Bias in LLM-as-a-Judge Causal Interventions in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation. Prior work has documented biases such as position, verbosity, and style preferences, but largely focuses on outcomes, leaving judge explanations underexplored. We instead ask whether LLM judges are cue-invariant, i.e., whether their rankings and explanations remain stable when non-evidential cues are perturbed while holding the underlying texts fixed. We introduce a suite of cue interventions (Blind, Truth, Flip, Placebo, Reveal-After) and tie-aware metrics that quantify outcome anchoring and rationale anchoring, including label-aligned rhetoric and explanation drift, alongside consistency and stereotype-intrusion checks. We design anchoring attacks using verbosity and confidence cues, and compare two mitigations: structured chain-of-thought prompting and PROOF-BEFORE-PREFERENCE (evidence lock, score, rank). Using a new dataset of 1,000 summaries from traditional extractive models and LLMs, we find substantial cue-anchored rationalization under label and placebo perturbations, while PROOF-BEFORE-PREFERENCE markedly improves cue invariance over baselines.

Abstract

LLM Evaluation

Explore similar work

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs

Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts