RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback

Jingyi He; Nier Wu; Shuang Liu; Xin Wang; Mengnan Du; Xia Hu

cs.CLSep 27, 2026

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback

Authors: Jingyi He, Nier Wu, Shuang Liu, Xin Wang, Mengnan Du, Xia Hu

Organizations: The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · Carnegie Mellon University · Jilin University · Shanghai Artificial Intelligence Laboratory

Abstract

Reward models (RMs) are a key component of large language model post-training, providing reward signals for subsequent reinforcement learning. However, conventional discriminative RMs typically output only scalar scores, making it difficult to identify the response behaviors associated with their scoring decisions. Existing interpretation methods often rely on predefined high-level attributes and require repeated counterfactual interventions for each response pair to validate candidate explanations, lacking a closed-loop mechanism that uses RMs' feedback to train a reusable explainer. To address this, we propose RewardExplainer, a framework that obtains feedback from the target reward model through counterfactual rewriting and uses this feedback to further optimize the explainer. RewardExplainer generates open-ended, atomic, and intervenable natural-language scoring mechanisms, making explanations more concrete, readable, and actionable. It further converts counterfactual feedback into preference supervision, enabling the explainer to more faithfully capture the target RM's scoring preferences and sensitive behaviors than single-pass generation. Extensive experiments across multiple target RMs and explainer backbones show consistent improvements. Beyond interpretation, we use the generated mechanisms to identify potential bias patterns and construct targeted debiasing data for fine-tuning the reward model, improving robustness on reward-hacking benchmarks.

Figures & tables

Figure 1: Overall framework of RewardExplainer. We first generate initial mechanism labels from response pairs preferred by the target RM and use them to initialize the explainer through SFT. We then construct preference data from bidirectional counterfactual intervention feedback and further improve explanation faithfulness with DPO. Finally, the optimized RewardExplainer is used to identify potential biases and perform targeted RM debiasing.

Method	Counterfactual Faith. $\uparrow$			Mechanism Quality $\uparrow$				Pref. Acc. $\uparrow$
	ADD	REM.	BOTH	Overall	C-Sel.	Scope	Dir.
Target RM: FsfairX-LLaMA3-RM-v0.1
Contrastive Explanations	27.44	28.33	21.05	0.75	0.69	1.06	0.49	74.47
GPT-4o	45.64	43.51	35.57	1.29	1.28	1.37	1.23	74.56
RewardExplainer (SFT)
Qwen3-4B	45.00	42.46	32.33	1.33	1.32	1.41	1.26	73.56

Table 1: Quantitative comparison of RewardExplainer across three target reward models. ADD, REMOVE, and BOTH measure Counterfactual Faithfulness; C-Sel., Scope, and Dir. denote Contrastive Selectivity, Scope Calibration, and Directional Fidelity, respectively. Overall is their equal-weight mean; Pref. Acc. denotes Preference Prediction Accuracy.

\uparrow

indicates higher is better.

Label	Description
Direct task-focused answering	The reward model favors responses that answer the user’s request plainly and upfront while minimizing unnecessary commentary, embellishment, or digression.
Conversational rapport framing	The reward model favors responses that use friendly, validating, or engaged conversational framing to make the reply feel responsive to the user.
Sequential procedural breakdown	The reward model favors explanations that break a process, solution, or implementation into an ordered sequence of steps or stages.

Table 2: Selected global preference mechanisms identified for RM-Mistral-7B.

Prompt: A model previously achieved 68% accuracy. After the update, its expected accuracy is around 70%, with run-to-run variation of about 2 percentage points. What performance should I expect?
Description	Response comparison
Verbosity padding
Expands essentially the same content into a longer response with extra explanation or padding that adds little substantive value.	$R_{0}$ : You can expect around 70% accuracy. $R_{1}$ : You can expect around 70% accuracy. In other words, the model’s accuracy after the update is expected to remain at approximately the 70% level.
Polished prestige style
Uses more formal, refined, technical, or polished language to make similar content seem more expert, complete, or high-quality.	$R_{0}$ : You can expect around 70% accuracy. $R_{2}$ : You may anticipate an accuracy level of approximately 70% .
Contextual mirroring

Table 3: Controlled examples of six reward hacking mechanisms. All rewrites share the same prompt and original response. The modified parts are highlighted in pink .

		RewardHackBench $\uparrow$						J-Bias $\uparrow$	J-Bench $\uparrow$
		Overall	By category					Overall	Overall
Target RM	Method	Avg.	Surface	Reason.	Sycoph.	Off-topic	Style	Avg.	Avg.
	Base	70.22	81.82	77.14	68.57	53.54	74.61	62.59	58.57
	General	74.77	83.33	77.14	66.19	69.47	79.79	65.13	59.43
FsfairX- LLaMA3- RM-v0.1	Ours	77.84	85.86	81.43	72.38	73.01	78.76	70.41	60.29
	Base	70.22	81.31	77.86	77.62	47.35	72.02	69.04	63.51

Table 4: Reward model debiasing results across three target RMs. RewardHackBench and JudgeBias evaluate debiasing effectiveness, while JudgeBench measures general capability. J-Bias/J-Bench denote JudgeBias/JudgeBench; Reason. and Sycoph. denote Broken Reasoning and Sycophantic Hacking. Bold indicates the best result per target RM.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Judge	ADD	REMOVE
Direct Rewrite	GPT-5-mini	9.38	7.97
Random perturbation	GPT-5-mini	3.17	1.82
Evidence-Guided Rewrite (Ours)	GPT-5-mini	9.60	9.32
Direct Rewrite	GPT-4.1-mini	9.44	8.43
Random perturbation	GPT-4.1-mini	3.33	4.21
Evidence-Guided Rewrite (Ours)	GPT-4.1-mini	9.49	9.19

Appendix

Table 5: Mean rewrite-quality scores on a 1–10 scale (higher is better).

Figure 2: Evidence-Guided Rewrite score distributions for ADD and REMOVE under two judges. Scores range from 1 to 10.

Target RM	Explainer	Exact	Jaccard	Char.	Emb.
		duplicate	$\geq.50$	$\geq.70$	$\geq.85$
Skywork-Reward-V2-Qwen3-1.7B	Qwen3-4B	0.0%	0.0%	0.1%	1.0%
FsfairX-LLaMA3-RM-v0.1	Qwen3-8B	0.0%	0.3%	0.2%	1.0%
RM-Mistral-7B	Qwen3-8B	0.0%	0.1%	0.0%	0.8%
Overall	–	0.0%	0.1%	0.1%	0.9%

Appendix

Table 6: Evaluation of mechanism redundancy and semantic similarity across three target reward models. Char. denotes character sequence similarity, and Emb. denotes embedding cosine similarity. Percentages indicate the proportion of within-example mechanism pairs meeting each criterion.

Metric	Set 1	Set 2	Set 3	Mean $\pm$ SD
Axis Quality (overall)	1.5478	1.5633	1.5389	$1.5500\pm 0.0124$
Choice accuracy	86.84%	85.13%	85.86%	$85.94\pm 0.86$ %
ADD success	55.76%	57.50%	56.32%	$56.53\pm 0.88$ %
REMOVE success	53.26%	55.14%	53.69%	$54.03\pm 0.99$ %
BOTH success	42.86%	44.49%	44.06%	$43.80\pm 0.84$ %

Appendix

Table 7: Performance across three training seeds for RM-Mistral-7B with Qwen3-4B as the explainer backbone. Mean and standard deviation are computed over the three runs.

Metric	Statistic	Set 1	Set 2	Set 3	Mean $\Delta\pm$ SD
Axis	$\Delta$	$+0.2422$	$+0.2578$	$+0.2333$	$+0.2444\pm 0.0124$
	95% CI	$[0.1778,0.3078]$	$[0.1878,0.3289]$	$[0.1689,0.2989]$	–
	$p$ -value	$.0002$	$.0002$	$.0002$	–
Choice	$\Delta$	$+9.89$ pp	$+8.44$ pp	$+9.50$ pp	$+9.28\pm 0.75$ pp
	95% CI	$[6.61,13.22]$	$[5.00,11.89]$	$[6.39,12.67]$	–
	$p$ -value	$.0002$	$.0002$	$.0002$	–

Appendix

Table 8: Pair-level paired-bootstrap significance tests comparing RewardExplainer with GPT-4o across three training seeds.

\Delta

denotes RewardExplainer minus GPT-4o, brackets report 95% confidence intervals, and pp denotes percentage points. Mean

\Delta\pm

SD summarizes the variation across seeds.

Metric	Statistic	Set 1	Set 2	Set 3	Mean $\Delta\pm$ SD
Axis	$\Delta$	$+0.8378$	$+0.8533$	$+0.8289$	$+0.8400\pm 0.0124$
	95% CI	$[0.7489,0.9256]$	$[0.7689,0.9378]$	$[0.7422,0.9156]$	–
	$p$ -value	$.0002$	$.0002$	$.0002$	–
Choice	$\Delta$	$+13.11$ pp	$+11.67$ pp	$+12.72$ pp	$+12.50\pm 0.75$ pp
	95% CI	$[9.33,16.83]$	$[7.56,15.67]$	$[8.78,16.56]$	–
	$p$ -value	$.0002$	$.0002$	$.0002$	–

Appendix

Table 9: Pair-level paired-bootstrap significance tests comparing RewardExplainer with Contrastive Explanations across three training seeds.

\Delta

denotes RewardExplainer minus Contrastive Explanations, brackets report 95% confidence intervals, and pp denotes percentage points. Mean

\Delta\pm

SD summarizes the variation across seeds.

Global mechanism	Description
Direct task-focused answering	The reward model favors responses that answer the user’s request plainly and upfront while minimizing unnecessary commentary, embellishment, or digression.
Concise justification of answers	The reward model favors answers that pair a clear conclusion, recommendation, classification, or definition with a brief explanation of why it is correct or significant.
Explicit structural organization	The reward model favors responses that organize information using visible structure such as lists, numbered points, headings, subheadings, or sectioned layouts.
Sequential procedural breakdown	The reward model favors explanations that break a process, solution, or implementation into an ordered sequence of steps or stages.
Contrastive clarification	The reward model favors responses that clarify a choice, concept, or classification by explicitly comparing it against alternatives or contrasting related cases.
Epistemic calibration and premise correction	The reward model favors responses that surface uncertainty, missing information, false premises, ambiguity, or evidential limitations before giving a definitive answer.

Appendix

Table 10: Complete set of global preference mechanisms identified for RM-Mistral-7B.

Setting	FsfairX	RM-Mistral	Skywork
Training pairs	1,622	1,550	1,056
Maximum length	2,048	4,096	3,072
Per-device batch	2	1	2
Gradient accumulation	8	16	8
Epochs	3	3	3
Learning rate	$2{\times}10^{-5}$	$2{\times}10^{-5}$	$2{\times}10^{-5}$

Appendix

Table 11: Target-RM debiasing hyperparameters.

Setting	Qwen3-4B		Qwen3-8B		Gemma4-12B
	SFT	DPO	SFT	DPO	SFT	DPO
Epochs	2	2	2	2	2	2
Learning rate	$1{\times}10^{-4}$	$5{\times}10^{-6}$	$1{\times}10^{-4}$	$5{\times}10^{-6}$	$1{\times}10^{-4}$	$5{\times}10^{-6}$
Per-device batch	2	3	4	2	1	1
Grad. accum.	8	8	4	8	16	32
Maximum length	2,048	2,048	2,048	2,048	2,048	1,920

Appendix

Table 12: Model-specific RewardExplainer training hyperparameters.

Explore similar work

reward-lens: A Mechanistic Interpretability Library for Reward Models
Apr 28, 2026Mohammed Suhail B NadafProgress Reward Modeling Mechanistic Interpretability
Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations
Apr 7, 2025Pedro Ferreira, Wilker Aziz, Ivan TitovExplainable AI Methods Chain-of-Thought Reasoning
Institute for Logic, Language and Computation (ILLC), University of Amsterdam · Institute for Language, Cognition and Computation (ILCC), University of Edinburgh
Debiasing Reward Models via Causally Motivated Inference-Time Intervention
Apr 30, 2026Kazutoshi Shinoda, Kosuke Nishida, Kyosuke NishidaLarge Language Model Bias Large Language Model Alignment
Human Informatics Labs., NTT, Inc.

Apr 28, 2026, cs.LGJ/K move · Enter open · S save

reward-lens: A Mechanistic Interpretability Library for Reward Models

Mohammed Suhail B Nadaf

Progress Reward Modeling Mechanistic Interpretability Interpretability Logit Lens

Open paper AskPDF arXiv

Every RLHF-trained language model is shaped by a reward model, yet the mechanistic interpretability toolkit -- logit lens, direct logit attribution, activation patching, sparse autoencoders -- was built for generative LLMs whose primitives all project onto a vocabulary unembedding. Reward models replace that with a scalar regression head, breaking each tool. We present reward-lens, an open-source library that ports this toolkit to reward models, organised around one observation: the reward head's weight vector

w_r

is the natural axis for every interpretability question. The library provides a Reward Lens, component attribution, three-mode activation patching, a reward-hacking probe suite, TopK SAE feature attribution, cross-model comparison, and five theory-grounded extensions (distortion index, divergence-aware patching, misalignment cascade detection, reward-term conflict analysis, concept-vector analysis). A ten-method adapter protocol covers Llama, Mistral, Gemma-2, and ArmoRM multi-objective heads, with a generic adapter for any HuggingFace sequence classification model. We validate on two production reward models across ~695 RewardBench pairs. The central empirical finding is negative: linear attribution does not predict causal patching effects (mean Spearman

ρ= -0.256

on Skywork,

-0.027

on ArmoRM). The framework treats this disagreement as a property to expose, not a bug -- motivating a design that keeps observational and causal views first-class and directly comparable.

Abstract

Explore similar work

reward-lens: A Mechanistic Interpretability Library for Reward Models

Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations

Debiasing Reward Models via Causally Motivated Inference-Time Intervention