EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models

J. Ben Tamo; Yuxing Lu; Benoit L. Marteau; Micky C. Nnamdi; May D. Wang

cs.CLMar 20, 2026

EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models

Authors: J. Ben Tamo, Yuxing Lu, Benoit L. Marteau, Micky C. Nnamdi, May D. Wang

Organizations: Georgia Institute of Technology · Pekin University

Abstract

Large Language Models (LLMs) are fluent but prone to hallucinations, producing answers that appear plausible yet are unsupported by available evidence. This failure is especially problematic in high-stakes domains where decisions must be justified by verifiable information. We introduce EvidenceRL, a reinforcement learning framework that enforces evidence adherence during training. EvidenceRL scores candidate responses for grounding (entailment with retrieved evidence and context) and correctness (agreement with reference answers) and optimizes the generator using Group Relative Policy Optimization (GRPO). We evaluate across two high-stakes domains, cardiac diagnosis and legal reasoning, where EvidenceRL consistently improves evidence grounding and faithfulness without sacrificing task accuracy. On cardiac diagnosis, F1@3 increases from 37.0 to 54.5 on Llama-3.2-3B while grounding ( $G_{\max}@3$ ) rises from 47.6 to 78.2; hallucinations drop nearly 5 $\times$ and evidence-supported diagnoses increase from 31.8% to 61.6%. On legal reasoning, EvidenceRL raises Faithfulness from 32.8% to 67.6% on Llama-3.1-8B, demonstrating consistent behavioral change across domains. Our code is open-sourced at https://github.com/Wizaaard/EvidenceRL.git.

Figures & tables

Figure 1: EvidenceRL aligns task accuracy with faithful evidence use across domains. Training uses GRPO with rewards for correctness ( $r_{c}$ ), format ( $r_{f}$ ), and evidence grounding ( $r_{g}$ ). Grounding is computed via a Focus–Then–Verify procedure: (1) focused (premise, hypothesis) pairs are constructed by combining an anchor context with individual evidence sections, and (2) each pair is scored by a frozen NLI cross-encoder.

Method	Backbone	F1@3 ( $\uparrow$ )	$G_{\mathrm{avg}}$ @3 ( $\uparrow$ )	$G_{\mathrm{max}}$ @3 ( $\uparrow$ )	EB ( $\uparrow$ )	H ( $\downarrow$ )	W ( $\downarrow$ )	LG ( $\downarrow$ )	F ( $\uparrow$ )
Reasoning Only	Llama-3.2-3B	37.0 $\pm$ 1.6	45.3 $\pm$ 3.5	47.6 $\pm$ 3.6	31.8%	11.6%	16.5%	6.2%	72.5%
	Llama-3.1-8B	38.2 $\pm$ 1.3	19.4 $\pm$ 3.6	21.3 $\pm$ 3.7	27.4%	17.7%	20.2%	11.5%	56.6%
	Llama-3.3-70B	51.3 $\pm$ 1.3	46.9 $\pm$ 3.0	49.4 $\pm$ 3.1	43.8%	8.5%	14.3%	9.3%	71.9%
	Gemma-3-4B	44.2 $\pm$ 1.4	34.3 $\pm$ 3.3	36.3 $\pm$ 3.4	35.1%	13.4%	19.2%	8.9%	66.0%
	Gemma-3-12B	46.5 $\pm$ 1.3	21.5 $\pm$ 3.2	23.5 $\pm$ 3.4	31.0%	14.4%	19.2%	14.3%	56.3%
	Gemma-3-27B	52.2 $\pm$ 1.3	37.1 $\pm$ 3.2	39.0 $\pm$ 3.4	39.1%	8.5%	15.5%	13.7%	63.1%

Table 1: Diagnostic performance and evidential reliability (

\pm

95% bootstrap CI). We report F1@3, average grounding (

G_{\mathrm{avg}}@3

), and a diagnostic taxonomy decomposing predictions into evidence-based correctness (EB), hallucinations (H), and lucky guesses (LG). Faithfulness (F) measures the fraction of correct predictions supported by evidence (Eq. 12 ). Green marks the best backbone within each method; yellow marks the overall best.

Method	Backbone	Acc. ( $\uparrow$ )	$G_{\text{avg}}$ ( $\uparrow$ )	$G_{\max}$ ( $\uparrow$ )	EB ( $\uparrow$ )	H ( $\downarrow$ )	W ( $\downarrow$ )	LG ( $\downarrow$ )	F ( $\uparrow$ )
Reasoning Only	Llama-3.2-3B	42.7 $\pm$ 9.0	-3.8 $\pm$ 7.2	5.3 $\pm$ 13.6	13.7%	15.4%	46.2%	9.4%	32.0%
	Llama-3.1-8B	57.3 $\pm$ 9.4	-1.7 $\pm$ 8.6	3.3 $\pm$ 12.2	18.8%	10.3%	51.3%	12.8%	32.8%
	Llama-3.3-70B	82.1 $\pm$ 6.8	13.9 $\pm$ 4.8	49.5 $\pm$ 12.4	50.4%	1.7%	30.8%	6.0%	61.5%
	Gemma-3-4B	47.9 $\pm$ 8.6	5.3 $\pm$ 6.6	14.5 $\pm$ 14.8	19.7%	10.3%	44.4%	6.8%	41.1%
	Gemma-3-12B	60.7 $\pm$ 8.5	4.2 $\pm$ 4.6	21.0 $\pm$ 15.3	22.2%	5.1%	43.6%	12.0%	36.6%
	Gemma-3-27B	65.8 $\pm$ 8.5	4.5 $\pm$ 4.4	20.2 $\pm$ 15.7	28.2%	7.7%	45.3%	7.7%	42.9%

Table 2: BarExam MBE: Accuracy, Evidence Grounding, and Diagnostic Taxonomy across backbone models. Wider confidence intervals reflect the smaller test(

n{=}173

) set compared to the medical domain (

n{=}1{,}000

Figure 2: At

\tau=0.80

, precision remains high for both approaches. Stable recall and high Cohen’s

\kappa

indicate a conservative reward signal, with no evidence of proxy hacking by GRPO models.

Figure 3: All five metrics shift in the same direction under both evaluators, confirming that grounding improvements reflect genuine evidence use rather than reward model overfitting.

Figure 4: SFT yields reasonable accuracy but weak grounding. GRPO with correctness reward (

r_{c}{+}r_{f}

) maximizes F1, while adding the grounding reward (

r_{g}

) substantially improves evidence attribution with only minor accuracy trade-offs.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Medical	BarExam
Embedding model	all-MiniLM-L6-v2
Embedding dim.	384
Index type	FAISS HNSW
Chunk size	320 tokens	N/A (pre-segmented)
Chunk overlap	64 tokens	N/A
Top- $k$	3	1 (gold) / 3 (FAISS)

Appendix

Table 3: Retrieval pipeline configuration.

Model	Family	Params	Inference
Llama-3.2-3B-It	Llama 3.2	3B	bf16, TP=2
Llama-3.1-8B-It	Llama 3.1	8B	bf16, TP=2
Llama-3.3-70B-It	Llama 3.3	70B	bf16, TP=4
Gemma-3-4B-IT	Gemma 3	4B	bf16, TP=2
Gemma-3-12B-IT	Gemma 3	12B	bf16, TP=2
Gemma-3-27B-IT	Gemma 3	27B	bf16, TP=2

Appendix

Table 4: Model backbones used in experiments. All models run in bf16 via vLLM for inference.

Parameter	Standard	Self-Consistency
Temperature	0.7	0.9
Top- $p$	0.9	0.9
Top- $k$	50	50
Repetition penalty	1.15	1.15
Max tokens	2,048 / 1,024 ∗	2,048
$n$ (samples per prompt)	1	10

Appendix

Table 5: vLLM inference configuration.

Parameter	Value
Epochs	3
Learning rate	$2\times 10^{-5}$
Scheduler	Cosine with 10% warmup
Per-device batch size	2
Gradient accumulation	8
Max sequence length	2,048 tokens

Appendix

Table 6: SFT training hyperparameters.

Parameter	Value
Epochs	2
Learning rate	$5\times 10^{-6}$
Scheduler	Cosine with 10% warmup
Per-device batch size	1
Gradient accumulation	8
DPO $\beta$	0.1

Appendix

Table 7: fDPO training hyperparameters. Uses TRL DPOTrainer with sigmoid loss on faithfulness preference pairs.

Parameter	Value
Epochs	2
Learning rate	$5\times 10^{-6}$
Scheduler	Cosine with 5% warmup
Gradient accumulation	8
Group size $G$	8 completions
Temperature	0.7 (medical); 0.7–1.0 (legal) $\dagger$

Appendix

Table 8: GRPO training hyperparameters. Implemented using TRL GRPOTrainer with vLLM colocate mode for on-policy generation.

Parameter	Value
Rank ( $r$ )	16
Alpha ( $\alpha$ )	32
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj,
	gate_proj, up_proj, down_proj
Bias	None

Appendix

Table 9: LoRA adapter configuration (shared across SFT, fDPO, and GRPO).

Figure 5: Training reward dynamics across model scales and objectives using MIMIC. We illustrate the training progress for the Llama-3.1-8B, Llama-3.2-3B, and the Gemma-3 series (4B, 12B, and 27B) Performance is evaluated across three primary reward components: (left) Format Reward (

r_{f}

) measuring adherence to structural constraints; (center) Accuracy Reward (

r_{c}

) assessing the correctness of generated responses; and (right) Grounding Reward (

\tilde{r}_{g}

) quantifying the extent to which outputs are supported by provided context. Larger model scales generally exhibit higher reward ceilings and more stable convergence across all metrics.

Figure 6: F1@3 vs.

G_{\max}

@3 across all model backbones and approaches. EvidenceRL points (blue) occupy the Pareto-optimal region—high accuracy and high grounding. SFT points (orange) collapse to near-zero grounding despite moderate accuracy. Self-RAG (green) clusters with or below zero-shot baselines.

Figure 7: Diagnostic taxonomy distribution for Llama-3.2-3B across all five approaches. Each bar decomposes top-3 predictions into six categories. Faithfulness (F) is annotated above each bar. EvidenceRL achieves the highest Evidence-Based rate (61.1%) and Faithfulness (87%), while Self-RAG has the lowest Faithfulness (60%) due to elevated Lucky Guess and Hallucination rates.

Figure 8: Per-patient grounding distributions (Llama-3.2-3B). Each violin shows the full distribution of

G_{\max}

@3 across patients. Diamonds indicate means; white lines indicate medians. EvidenceRL concentrates mass near 1.0 (median=0.98, 81% of patients above the grounding threshold), while other approaches show broad, bimodal distributions.

Approach	Top-1 Dx	$G_{\max}$	Reasoning (excerpted)
EvidenceRL	Myocardial Ischemia	+0.93	“The presence of a significant LAD lesion with 90% proximal occlusion , combined with the patient’s recent stress test showing antero-apical perfusion defect , strongly suggests intermittent myocardial ischemia. The patient’s elevated troponin level (4.67) and reduced ejection fraction (LVEF = 42%) support the diagnosis.”
SFT	NSTEMI	$-$ 0.35	“The initial presentation included elevated Troponins (4.67 ng/mL) , indicative of myocardial necrosis…a previously documented stress test showing an apical perfusion defect followed by development of apical thrombi…warranting immediate evaluation.”
Zero-Shot	CAD	+0.14	“The patient presents with a history of NSTEMI and subsequent stress test showing ant/apical perfusion defects …The presence of 90% LAD lesion, 50% CX , and successful stenting of the LAD support this diagnosis.”
Self-RAG	CAD	$-$ 0.34	“The presence of a 90% proximal lesion in the LAD explains the patient’s angina…The physical exam shows elevated blood pressure (SBP 153 mmHg) …Criteria from Evidence 1 and 2 support the presence of coronary artery disease.”
SC	CAD	+0.08	(Consensus from $N$ =10 samples; reasoning from the highest-ranked generation.)

Appendix

Table 10: Reasoning comparison for the top-ranked diagnosis of patient #29017807 across all five approaches. Clinical values cited from the patient record are underlined .

Approach	Correct	$G_{\mathrm{avg}}$ @5	Combined
EvidenceRL	5/5	+0.758	0.879
SFT	4/5	$-$ 0.869	$-$ 0.034
SC	4/5	+0.084	0.442
Zero-Shot	3/5	$-$ 0.694	$-$ 0.047
Self-RAG	3/5	$-$ 0.861	$-$ 0.130

Appendix

Table 11: Case study summary for patient #29017807 (Llama-3.2-3B).

Explore similar work

Jul 1, 2026

FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning

Faithful reasoning is essential in medicine, where clinical decisions require transparent justification grounded in reliable evidence. Current medical LLMs either lack active access to evidence or use retrieved evidence without supervising how it should be appraised and applied during reasoning. To address this, we formalize evidence-based medicine principles as process-level criteria and introduce FaithMed, a framework that combines clinician-designed, automatically refined rubrics with reinforcement learning using step-level process reward assignment and advantage grouping. Across seven medical benchmarks, FaithMed improves over agentic-search baselines (+9% on average) and outcome-only RL (+5.8%), while raising average evidence-based medicine rubric scores over agentic-search Qwen3 baselines (+15.5%). This work demonstrates that explicit step-level supervision can improve both task success and the faithfulness of the reasoning process. Code is available at https://github.com/cxcscmu/FaithMed.

Zhiyun Zhang, Liwen Sun, Xiang Qian +1

Carnegie Mellon University · Stanford University School of Medicine · Xlue

RL for Language Model Reasoning Process Supervision Medical Reasoning Medical LLMs

Ask PDF

Sep 29, 2026

Learning to Prove, Not Just to Answer: Reinforcement Learning from Formal Verification for Natural-Language Logical Reasoning

Large language models (LLMs) are increasingly deployed for natural-language logical reasoning, where the final answer is easy to check but the proof behind it is not. In natural-language logical reasoning, an intermediate conclusion should follow from its premises, and the resulting derivation should support the final answer. Existing methods lack machine-checkable verification of intermediate conclusions and answer-supporting proof dependencies, so they may assign credit to invalid or answer-irrelevant steps. We propose Proof-R1, an RL framework from formal verification that trains LLMs to construct verifiable proofs for natural-language logical reasoning. Proof-R1 admits a generated conclusion into the verified proof state only when the corresponding reasoning action satisfies the proof obligations through UNSAT-based machine-checkable formal verification. Proof-R1 also recovers the answer-supporting dependency closure to trace the proof structure of the final answer and align outcome credit with the proof dependencies. Experiments demonstrate that Proof-R1 improves answer accuracy across three logical reasoning benchmarks and four backbone models and outperforms training-free agents and training-based methods in terms of reasoning-process verifiability.

Qili Zhang, Qianren Mao, Hanze Cai +11

Zhongguancun Laboratory · Beihang University · Nanyang Technological University +1

Logical Reasoning RL for Language Model Reasoning

Ask PDF

Apr 24, 2026

Learning Evidence Highlighting for Frozen LLMs

Large Language Models (LLMs) can reason well, yet often miss decisive evidence when it is buried in long, noisy contexts. We introduce HiLight, an Evidence Emphasis framework that decouples evidence selection from reasoning for frozen LLM solvers. HiLight avoids compressing or rewriting the input, which can discard or distort evidence, by training a lightweight Emphasis Actor to insert minimal highlight tags around pivotal spans in the unaltered context. A frozen Solver then performs downstream reasoning on the emphasized input. We cast highlighting as a weakly supervised decision-making problem and optimize the Actor with reinforcement learning using only the Solver's task reward, requiring no evidence labels and no access to or modification of the Solver. Across sequential recommendation and long-context question answering, HiLight consistently improves performance over strong prompt-based and automated prompt-optimization baselines. The learned emphasis policy transfers zero-shot to both smaller and larger unseen Solver families, including an API-based Solver, suggesting that the Actor captures genuine, reusable evidence structure rather than overfitting to a single backbone.

Shaoang Li, Yanhang Shi, Yufei Li +10

Stony Brook University · Meta AI

Long-Context QA LLM Prompting

Ask PDF