Calibrating LLM Explanations to Human Decisions with Reinforcement Learning

Dong Shu; Ziyang Guo; Negar Kamali; Matthew Groh; Jessica Hullman

cs.HCOct 12, 2026

Calibrating LLM Explanations to Human Decisions with Reinforcement Learning

Authors: Dong Shu, Ziyang Guo, Negar Kamali, Matthew Groh, Jessica Hullman

Organizations: Northwestern University

Abstract

Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce HD-RL, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions. We apply this framework to AI-generated text detection. In our initial data collection phase, we observed that while LLMs possess strong autonomous detection capabilities, the explanations they generate for users yield near-random human accuracy (51.3%). By applying the HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that improve human performance, achieving a GLMM-estimated human accuracy of 63.7% in a randomized between-subjects experiment with repeated measures and outperforming RLVR, RLAIF, and other baselines by over 8%.

Figures & tables

Figure 1: The overall architecture of the HD-RL framework. Modules marked with a fire icon represent trainable parameters, whereas those marked with a snow icon denote frozen models.

Figure 2: Human Decision Data and Decision Simulator Training Report.

Figure 3: GLMM-Estimated Human Accuracy across Experimental Conditions. This plot displays the model-derived human accuracy estimates for each alignment framework, complete with 95% Confidence Intervals. For each condition, the aggregate performance (purple) is staggered alongside the disaggregated performance on human-written texts (orange) and AI-generated texts (blue).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: LLM Decision Baseline Simulator Calibration and VR-GRPO Training Report. (a) Data statistics for the LLM-generated synthetic decisions. (b) Training and testing performance of the simulator, demonstrating near-optimal convergence on the LLM’s decision distribution. (c) VR-GRPO training loss over time, along with the test accuracy of 85.7%.

Figure 5: Analysis of the text dataset, human decisions, and explanations. (a) Text length distributions for human and AI texts demonstrate structural parity. (b) Human decision accuracy broken down by explanation source reveals a severe skew toward human text detection. (c) The percentage of explanations containing vocabulary appropriate for the ground-truth text label. (d) The LLM judge exhibits a massive bias toward human-focused explanations regardless of the ground-truth label. (e) Human decision accuracy on AI texts demonstrates that AI-focused explanations are required for high human accuracy.

Method	Trials	Participants	Texts
HD-RL	2,488	230	678
LLM_decision	1,572	136	678
DPO	2,113	208	678
VR_GRPO	1,282	119	678
Total	7,455	693	678

Appendix

Table 1: Distribution of Human Trials in the Deployment Experiment.

Term	Estimate	95% CI	$p$ -value
(Intercept) — HD-RL , Human Truth	0.688	[0.522, 0.854]	$<.001$
Condition: LLM_decision	-0.206	[-0.462, 0.049]	.113
Condition: DPO-RLAIF	-0.380	[-0.611, -0.149]	.001
Condition: VR-GRPO	-0.243	[-0.508, 0.022]	.073
Ground Truth: AI	-0.253	[-0.435, -0.071]	.006
Trial Number (centered)	0.009	[0.005, 0.013]	$<.001$

Appendix

Table 2: Fixed-effect coefficients from the joint Generalized Linear Mixed Model (GLMM) on the log-odds scale.

Condition	Ground Truth	$n$	Raw Acc.	Model Acc.	95% CI
HD-RL	Human	1,288	68.0%	66.6%	[62.8%, 70.1%]
LLM_decision	Human	802	61.2%	61.8%	[57.0%, 66.4%]
DPO-RLAIF	Human	1,084	57.7%	57.6%	[53.5%, 61.7%]
VR-GRPO	Human	669	59.0%	61.0%	[55.8%, 65.9%]
HD-RL	AI	1,200	63.0%	60.7%	[56.6%, 64.6%]
LLM_decision	AI	770	49.1%	48.2%	[43.3%, 53.1%]

Appendix

Table 3: Estimated marginal accuracies derived from the joint GLMM compared to raw empirical accuracies, disaggregated by experimental condition and text ground truth.

Condition Contrast	Overall Est. Acc.	Log-Odds Contrast	$p$ -value
HD-RL (Reference)	63.7%	–	–
LLM_decision vs. HD-RL	55.1%	-0.357	.001
DPO-RLAIF vs. HD-RL	51.0%	-0.522	$<.001$
VR-GRPO vs. HD-RL	54.6%	-0.376	.001

Appendix

Table 4: Overall estimated marginal accuracies and treatment contrasts, averaged across both AI and Human ground-truth distributions.

Condition	Log-odds Gap	95% CI	Odds Ratio (Human vs AI)	$p$ -value
HD-RL	0.253	[0.071, 0.435]	1.29 $\times$	.006
LLM_decision	0.554	[0.340, 0.767]	1.74 $\times$	$<.001$
DPO-RLAIF	0.538	[0.354, 0.722]	1.71 $\times$	$<.001$
VR-GRPO	0.520	[0.286, 0.754]	1.68 $\times$	$<.001$

Appendix

Table 5: Differential accuracy contrasts showing the performance gap within each condition (calculated as Human minus AI). A larger odds ratio indicates a more severe performance collapse when participants are tasked with identifying AI-generated texts.

Explainable Artificial Intelligence

Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.

See all 320 papers

Human Decision-Making with Persuasive and Narrative LLM Explanations
May 22, 2026Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash +1Explainable Artificial Intelligence Human-AI Decision Making
DEVCOM Army Research Laboratory Adelphi, MD 20783 · University of Texas at Dallas Richardson, TX 75080 · Virginia Polytechnic Institute and State University Blacksburg, VA 24060
Training Large Language Models for Self-Explanation Faithfulness
Jul 23, 2026Yeoktatt Cheah, María Pérez-Ortiz, Noah Y. Siegel +1Faithfulness of Language Model Explanations LLM Interpretability
Centre for AI, Department of Computer Science University College London · Imperial College London
All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models
Jul 10, 2026Pan LiExplainable Artificial Intelligence LLM Interpretability
Scheller College of Business, Georgia Tech.

May 22, 2026J/K move · Enter open · S save

Human Decision-Making with Persuasive and Narrative LLM Explanations

Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash, Murat Kantarcioglu

DEVCOM Army Research Laboratory Adelphi, MD 20783 · University of Texas at Dallas Richardson, TX 75080 · Virginia Polytechnic Institute and State University Blacksburg, VA 24060

Explainable Artificial Intelligence Human-AI Decision Making Explainability Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have the potential to aid and improve human decision-making in classification tasks, not only by providing fairly accurate predictions, but also in their ability to generate cogent narrative explanations of those predictions. Prior work has demonstrated that people generally find AI narrative explanations to be understandable, trustworthy, and convincing for changing beliefs and opinions; however, less is known about the impact of narrative explanations on objective human decision-making performance. Here we conduct a large-scale human behavioral experiment to evaluate decision-making performance with LLM-generated narrative explanations of varying persuasiveness. We found the degree of persuasiveness, or lack thereof, for LLM-based explanations did not meaningfully impact decision accuracy over a simple AI prediction alone, in agreement with typical results with explainable AI based on feature importance. We found evidence that narratives increased reliance on AI, but both when the AI prediction was correct and incorrect. Exploratory analyses also indicated that the more persuasive narratives may have had a detrimental effect on decision response times and the ability to discriminate between a correct and incorrect AI prediction. Overall, this work indicates that including narrative explanations with AI predictions may involve tradeoffs for decision-making performance, and more work is needed to determine how and when narrative explanations impact human decision-making.

Abstract

Explainable Artificial Intelligence

Explore similar work

Human Decision-Making with Persuasive and Narrative LLM Explanations

Training Large Language Models for Self-Explanation Faithfulness

All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models