AI systems increasingly shape how individuals access and evaluate information, intensifying concerns about misinformation. Computational simulation offers a controlled complement to direct empirical study, with fine-tuned LLMs enabling powerful agent-based simulations. However, fine-tuning often induces broad behavioral shifts including bias drift, degenerate heuristics, and changes in baseline task behavior, which undermine experimental validity. The central challenge is inducing localized decision shifts while preserving the model's aggregate behavior outside the targeted intervention. We study this through a stability analysis of supervised fine-tuning, attenuating loss on selected training slices to shift false-claim acceptance. We introduce FrameRef, a large-scale dataset of semantically equivalent claims with controlled surface framings across five dimensions (Authoritative, Consensus, Emotional, Sensationalist, Prestige), with human validation confirming systematic framing effects on claim acceptance. We present a methodology for identifying model-specific stability regimes in which targeted error shifts can be induced while preserving aggregate accuracy and calibration, and demonstrate its application to selected language models. A sequential exposure task over FrameRef shows that small framing-conditioned shifts compound into substantially different cumulative outcomes under feedback, but largely disappear when feedback is removed, confirming that interventions modify specific decision tendencies rather than global performance. We release code, LoRA adapters, and data at https://github.com/infosenselab/framing-sensitive-ft.
Figures & tables
Figure 1: Agents assign framing-dependent scores to the same underlying claim.
Framing
Claim
Original
Stevie Ray Vaughan was born in Dallas, Texas.
Authoritative
According to a widely recognized authority on historical figures, Stevie Ray Vaughan was born in Dallas, Texas.
Consensus
It is widely acknowledged that Stevie Ray Vaughan was born in Dallas, Texas.
Emotional
Texas is the birthplace of the legendary blues guitarist Stevie Ray Vaughan.
Sensationalist
Blasting onto the music scene, blues legend Stevie Ray Vaughan was born in the vibrant city of Dallas, Texas!
Prestige
A renowned blues guitarist, Stevie Ray Vaughan, hailed from the city of Dallas, Texas.
Table 1: Sample original & framed claims.
Fam. →
{1,2}
{3}
{4,5}
Frame ↓
FAcce
Conf.
pVal
FAcce
Conf.
pVal
FAcce
Conf.
pVal
Original
0.52
1.53
0.70
0.59
3.16
0.38
0.66
4.46
0.02
Auth.
0.50
1.87
1.00
0.40
3.16
0.23
0.52
4.29
0.89
Consensus
0.44
1.72
0.24
0.57
3.37
0.50
0.64
4.44
0.07
Emotional
0.52
1.77
0.65
0.52
2.89
1.00
0.78
4.59
0.00
Prestige
0.57
1.66
0.19
0.66
3.39
0.07
0.73
4.53
0.00
Table 2: Human evaluation false-claim acceptance and confidence by familiarity level. P-values test acceptance vs. chance (0.5). Bold: p<0.05 ; underline : p<0.10 .
Figure 2: System diagram describing the task workflow.
Train
BAcc
TNR
%SUPP
MSPR
Conf (C/I)
Llama-3.1-8B-Instruct
0.554
0.442
0.622
0.558
0.887/0.864
Llama-3.2-3B-Instruct
0.568
0.556
0.525
0.440
0.853/0.830
Llama-3.3-70B-Instruct
0.586
0.442
0.660
0.547
0.941/0.904
Qwen2.5-7B-Instruct
0.575
0.360
0.729
0.625
0.964/0.942
Qwen2.5-14B-Instruct
0.541
0.795
0.253
0.206
0.946/0.956
Mistral-7B-Instruct-v0.3
0.559
0.191
0.879
0.785
0.946/0.937
Table 3: Baseline performance with no SFT.
Train
BAcc
TNR
%SUPP
MSPR
Llama-3.1-8B-Instruct
No SFT
0.554
0.442
0.622
0.558
1k
0.540
0.449
0.598
0.537
10k
0.537
0.502
0.542
0.491
15k
0.521
0.449
0.577
0.515
25k
0.556
0.343
0.723
0.642
Table 4: Performance as training samples increase ( α=1 for all models).
Figure 3: Variability across α values (Authoritative Framing).
Framing
BAcc
Mean Conf (C/I)
Original
0.536
0.841/0.805
Authoritative
0.541
0.860/0.820
Consensus
0.542
0.846/0.813
Prestige
0.542
0.848/0.812
Emotional
0.529
0.843/0.801
Sensationalist
0.551
0.849/0.824
Table 5: Cumulative score calibration results using Llama-3.1-8B-Instruct using α=0.3 and 15k training samples (three-model average per framing).
Model →
Baseline
Auth.
Consensus
Emotional
Prestige
Sensat.
Random
Frame ↓
BAcc
MSPR
BAcc
MSPR
BAcc
MSPR
BAcc
MSPR
BAcc
MSPR
BAcc
MSPR
BAcc
MSPR
Original
0.566
0.529
0.553
0.585
0.562
0.543
0.521
0.502
0.569
0.552
0.582
0.564
0.572
0.628
Auth.
0.540
0.569
0.550
0.622
0.555
0.583
0.559
0.546
0.542
0.594
0.542
0.600
0.569
0.652
Consensus
0.547
0.618
0.544
0.672
0.533
0.636
0.517
0.596
0.545
0.644
0.527
0.652
0.548
0.733
Emotional
0.525
0.592
0.542
0.639
0.533
0.610
0.519
0.573
0.529
0.620
0.539
0.625
0.575
0.605
Prestige
0.526
0.495
0.537
0.544
0.540
0.513
0.548
0.471
0.545
0.519
0.551
0.529
0.552
0.703
Table 6: Llama-3.1-8B-Instruct diagnostic evaluation results. Models’ target framings are underlined for reference. Bold MSPR indicates increased credulity vs. baseline ( α=1 ).
Framing
Feedback
No Feedback
Lin
Log
Lin
Log
Baseline
7.38
-128.77
16.04
-110.55
Auth.
2.69
-147.91
17.47
-115.07
Consensus
5.56
-136.69
16.05
-115.04
Prestige
5.11
-137.59
17.25
-111.56
Sensat.
3.86
-144.60
17.97
-114.48
Table 7: Llama-8B cumulative score results with and without feedback (3-model averages; 100 trajectories each).
Measure
Framing
Feedback
No Feedback
Orig
Auth
Cons
Emot
Prest
Sens
Orig
Auth
Cons
Emot
Prest
Sens
Encounters ( ×103 )
Baseline
1.53
1.78
1.61
1.69
1.78
1.61
1.66
1.66
1.70
1.63
1.65
1.70
Auth.
1.47
1.77
1.61
1.68
1.81
1.66
1.68
1.64
1.70
1.66
1.68
1.64
Consensus
1.54
1.78
1.59
1.69
1.81
1.60
1.68
1.66
1.69
1.65
1.66
1.66
Prestige
1.53
1.81
1.59
1.65
1.78
1.65
1.69
1.65
1.66
1.66
1.65
1.70
Sensat.
1.52
1.78
1.57
1.67
1.82
1.64
1.65
1.67
1.65
1.64
1.69
1.71
Table 8: Llama-8B feedback vs no-feedback ablation results. Three-models per framing type averages.
Figure 4: Llama-8B average cumulative score under linear (left) and log (right) scoring. Linear score: higher is better. Log score: closer to 0 is better. Three models per framing type, 100 trajectories each.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Framing
Verif. Failed
Verif. Passed
Pass Rate (%)
Average Tokens
Original
NA
191,817
NA
21.4
Authoritative
5,430
186,387
97.2
28.0
Consensus
36,785
155,032
80.8
25.9
Emotional
10,712
181,105
94.4
30.2
Prestige
16,755
175,062
91.3
28.5
Sensationalist
7,480
184,337
96.1
36.9
Appendix
Table B.1: Dataset statistics. 191,817 claims were generated for each framing variant.
Field
Description
Raw
Split
claim_id
Unique claim ID.
✓
✓
claim_text
Original claim text.
✓
×
true_label
SUPPORTS/REFUTES.
✓
✓
restated_claim
Reformulated claim.
✓
✓
framing_type
Framing style.
✓
✓
verification_passed
Verification success Bool.
✓
×
Appendix
Table B.2: Dataset raw output and train, dev, and test splits schema.
Figure C.1: Introduction screen.
Figure C.2: Judgment question.
Figure C.3: Confidence question.
Figure C.4: Familiarity question.
Familiarity band
n
Acce (REF)
Acce (SUP)
Gap
Conf. (REF)
Conf. (SUP)
Low (1–2)
1,341
49.4
59.7
10.3
1.7
1.8
Medium (3)
417
44.7
74.4
29.8
3.2
3.2
High (4–5)
642
34.3
87.1
52.8
4.4
4.5
Overall
2,400
45.0
70.4
25.4
2.6
2.8
Appendix
Figure C.5: Human evaluation accuracy and confidence by familiarity band.
Table D.3: Cumulative score calibration results using Mistral-7B-Instruct-v0.3 with 15k training samples (three-model average per framing).
Model →
Baseline
Auth.
Consensus
Emotional
Prestige
Sensat.
Framing ↓
BAcc.
MSPR
BAcc.
MSPR
BAcc.
MSPR
BAcc.
MSPR
BAcc.
MSPR
BAcc.
MSPR
Original
0.583
0.613
0.575
0.620
0.584
0.572
0.586
0.529
0.591
0.585
0.594
0.589
Authoritative
0.564
0.653
0.569
0.663
0.576
0.595
0.564
0.555
0.577
0.612
0.568
0.619
Consensus
0.554
0.708
0.534
0.719
0.535
0.683
0.542
0.615
0.531
0.684
0.542
0.677
Emotional
0.548
0.688
0.550
0.691
0.547
0.626
0.553
0.598
0.535
0.640
0.553
0.668
Prestige
0.544
0.738
0.535
0.728
0.536
0.658
0.531
0.623
0.538
0.670
0.539
0.697
Appendix
Table D.4: Mistral-7B-Instruct-v0.3 Diagnostic evaluation results. Models’ target framings are underlined for reference.
Type
Framing
CS
SD
CI 95L
CI 95U
Feedback
lin
Baseline
7.38
12.11
5.99
8.78
lin
Auth.
2.69
13.88
1.13
4.26
lin
Consensus
5.56
12.06
4.21
6.91
lin
Prestige
5.11
12.69
3.68
6.54
lin
Sensat.
3.86
13.33
2.32
5.44
Appendix
Table E.5: Llama-8B cumulative score results with and without feedback statistical details (3-model averages; 100 trajectories each).
We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations. Large language models (LLMs) often change their responses to subtle rephrasings that align with an implied stance by users. This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts, and the consequences are highly costly in high-stakes domains. Because in the realistic scenarios, both expert practitioners and non-expert users frequently ask LLMs questions containing incomplete or misleading assumptions, models are highly susceptible to those framings. To test this, we inject the framing bias across three nested levels: a framing-biased question phrasing (root), an injected framing-biased premise prepended to a neutral question (propositional), and a premise paired with a framing-biased question (global). Evaluating nine open models (3.8B-70B) across four families, we find that strong per-variant accuracy does not guarantee the robustness across differently phrased questions under the fixed factual information.
Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo +10
Department of Computer Science, University of Oxford · Oxford Internet Institute, University of Oxford · Thomson Reuters +7
Correcting health misinformation in dialogue requires more than producing a factual rebuttal: users differ in what they know, what they believe, and what they need to hear, so an effective intervention often depends on first asking the right clarifying question. Yet existing methods either respond immediately or probe indiscriminately, treating clarification as either unnecessary or always beneficial. We propose Reward-Optimized Probe-and-Respond (RO-PnR), a framework that learns when asking is worth its cost. At each turn, RO-PnR chooses between probing for more information and committing to a final correction, guided by a turn-level reward that weighs the expected gain from probing against its interaction cost. To capture how user heterogeneity affects probing value, we model each simulated user with a latent state along health literacy and belief commitment. Experiments show that RO-PnR achieves the highest cost-adjusted utility across three health-misinformation datasets and three base models, using 30% fewer turns than always-probe baselines.
Misinformation is a growing societal threat, and susceptibility to misinformative claims varies across demographic groups due to differences in underlying beliefs. As Large Language Models (LLMs) are increasingly used to simulate human behaviors, we investigate whether they can simulate demographic misinformation susceptibility, treating beliefs as a primary driving factor. We introduce BeliefSim, a simulation framework that constructs demographic belief profiles using psychology-informed misinformation taxonomies and survey priors. We study prompt-based conditioning and post-training adaptation, and conduct a multi-fold evaluation using: (i) susceptibility alignment and (ii) counterfactual demographic sensitivity. Across both datasets and modeling strategies, we show that beliefs provide a strong prior for simulating misinformation susceptibility, with alignment up to 92%.
Angana Borah, Zohaib Khan, Rada Mihalcea +1
University of Michigan - Ann Arbor, USA · Texas State University