cs.CLDec 18, 2025

From Evidence to Judgment: Trace-Augmented Reasoning under Conflicting Evidence

Authors: Shubham Mishra, Atharv Sharma, Dhruv Kumar, Harsh Sharma

Organizations: Birla Institute of Technology and Science, Pilani · Carnegie Mellon University, Pittsburgh

Abstract

Retrieved documents often disagree, contain outdated information, or fail to support any answer, and retrieval-augmented generation (RAG) systems have no reliable way to decide which documents to trust. We treat this as a reasoning problem and test it separately from answer generation. We release 943 training and validation examples and a disjoint 736-example benchmark, and fine-tune four language models to write a structured think trace before answering: the model judges each document, infers the conflict pattern, and writes a grounded answer or a justified refusal. Our evaluation framework, CATS (Conflict-Aware Trust Score), scores grounded refusal, behavioural adherence, factual grounding, and single-truth recall; a human study validates factual grounding and single-truth recall (kappa=0.78 and 0.79). The think trace improves the answer/refuse decision: under all three prompts, untuned models score at most 0.429 Refusal F1 (F1 with refusal as the positive class) and trace-tuned models at least 0.826. On the answerable examples a model answers, factual grounding under the 40-word prompt is at most 0.11 for untuned and 0.86-0.88 for trace-tuned models. Cutting the prompt from 1,965 words to 40 leaves trace-tuned scores within the benchmark's measurement noise (+/-0.034). Answer-only fine-tuning trails trace supervision on every metric under the minimal prompt.

Figures & tables

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation

    Sep 24, 2026Zhengchen Huang, Yundong Sun, Minrui Song +4RAG RobustnessRetrieval-Augmented Generation

  2. RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

    Oct 8, 2026Shunyuan Zhou, Hao Chen, Tianyu Wang +3Retrieval-Augmented GenerationRAG Evaluation

  3. EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning

    Jul 11, 2026S M Asif Hossain, Ruksat Khan Shayoni, M. F. MridhaRetrieval-Augmented GenerationEvidential Deep Learning