Small variations in prompt characteristics can meaningfully alter the semantics, tone, and contextual assumptions of large language model outputs. Such characteristics can involve group variables such as demographics or other sensitive latent attributes. Sensitivity to such input variables is undesirable when demographic information is causally irrelevant to the task, but might still be relevant for generation of semantically coherent responses. This makes the task of enforcing counterfactual invariance while preserving model utility difficult. To study this problem, we first show that standard model alignment recipes (such as reinforcement learning) increase differences in response quality across demographics, while improving overall response quality, ultimately amplifying what is known as counterfactual bias. To address this, we develop a policy gradient technique involving a group-level reward function that operates on matched pairs of counterfactual prompts. For completeness, we build a dataset of controlled counterfactual prompt pairs and design an evaluation that separately measures cases where demographic information is causally relevant to the answer. Finally, we validate our findings on several fairness benchmarks. We demonstrate that our methods reduce counterfactual differences while preserving overall response quality and that the models continue to adapt appropriately when such sensitive information is relevant.
Figures & tables
Figure 1: Sensitive context such as demographics can subtly affect the quality of the open-ended response. Such dependence might be relevant in certain instances, but actively harmful in other situations and lead to diminished response quality or stereotyping. This paper studies the amplification of such harms as a side effect of training language models with reinforcement learning.
Figure 2: Overview of the proposed objectives for a pair of invariant prompts (x,x′) : The central idea in our paper involves enforcing invariance by encouraging the reward distributions across counterfactuals to be similar. Towards this goal, a natural choice involves matching the first moment , that is the expectation of the rewards.
Figure 3: Bias and marginal reward across methods: Across DiscrimEval and Difference Awareness, we find that helpfulness-only RL with vanilla GRPO amplifies counterfactual bias. Our proposed methods mitigate this amplification while improving marginal reward.
Figure 4: Invariant rubric scores of the proposed training methods versus vanilla RL: We find that although vanilla GRPO improves marginal reward, it lowers every rubric score. Our proposed methods retain higher scores throughout training.
Figure 5: Reward distributions under the same prompt: For each group, we score two sets of responses under that group’s prompt, the responses generated for the group and the responses generated for its counterpart after swapping the demographic references. Vanilla GRPO separates the distributions relative to the initializer, whereas Transformed Reward Expectation Matching keeps them aligned.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Mean Δ by sampling arm
Axis
Mean Δ
Mean ∣Δ∣
Favored rate
White/male
Black/female
Race
+0.243
0.508
69.0%
+0.331
+0.153
Gender
+0.236
0.437
73.8%
+0.379
+0.088
Appendix
Table 1: Reward differences for identical responses under demographic prompt swaps. The favored rate is the fraction of comparisons in which the white or male prompt receives the higher reward. The last two columns split responses by the prompt under which they were sampled. Rewards are on the native reward-model scale.
Input
Output of τ
The female patient should schedule a follow-up. She can call her doctor tomorrow.
The male patient should schedule a follow-up. He can call his doctor tomorrow.
I gave her the report, and her manager approved it.
I gave him the report, and his manager approved it.
His advice to his son was to trust himself. The idea was his alone.
Her advice to her daughter was to trust herself. The idea was hers alone.
As a Black entrepreneur, he should compare loan offers before Black Friday.
As a White entrepreneur, he should compare loan offers before Black Friday.
He wrote “my father was a man of few words” in the email to Mr. Smith.
She wrote “my father was a man of few words” in the email to Ms. Smith.
Appendix
Table 2: Examples of τ applied to responses.
Source family
Invariant
Dependent
Agnostic
Demographic-free
Anthropic Helpful Base
575
161
213
377
HealthBench
881
87
176
38
HelpSteer2
281
49
77
310
Wikidata People
0
82
0
0
Total
1,737
379
466
725
Appendix
Table 3: Composition of the dataset by source family and relation class. The counterfactual pairs comprise 1,636 race pairs and 946 gender pairs.
Class
Desired behavior
Example
Invariant
Preserve relevant content and tone.
Cooking instructions for a black/white user.
Dependent
Use demographic context when relevant.
Notable female/male filmmakers.
Agnostic
Accept either valid strategy.
Advice before a blind date.
Demographic-free
Respond correctly and helpfully.
What is 5G?
Appendix
Table 4: Desired behavior across counterfactual-pair relations and the demographic-free class.
Training element
Setting
Optimizer
Adafactor ( Shazeer and Stern, 2018 ) with relative-step scheduling disabled, learning rate 5×10−6 , 30 warmup updates followed by a constant rate, zero weight decay, and gradient clipping at 1.0
GRPO
KL coefficient β=10−3 , clipping coefficient ϵclip=0.2 , and rewards standardized within each prompt group
Group size
responses per prompt
Training decoding
Temperature 1.0, top- p=1.0 , and top- k=0
Compute
Each training run takes about 30 hours per fold on 3 GPUs with 96 GB of memory each
Appendix
Table 5: Training settings shared across conditions. A condition changes only its training data or explicitly named objective.
Figure 6: Effect of α on Transformed Reward Expectation Matching under invariant-only training.
Figure 7: Effect of α on Transformed Reward Expectation Matching under mixed-dataset training, evaluated on invariant pairs.
Figure 8: Effect of α on Transformed Reward Expectation Matching under mixed-dataset training, evaluated on dependent pairs.
Data and rubric
All three exact
All within one
Mean pairwise κw
Invariant
81.0%
85.3%
0.260
Dependent
72.4%
82.5%
0.386
Agnostic, invariant rubric
80.3%
84.5%
0.284
Agnostic, dependent rubric
65.2%
77.3%
0.339
Appendix
Table 6: Inter-judge agreement on the shared initializer. Percentages and quadratic-weighted Cohen’s κ are averaged across criteria in each row. “All within one” means the three ratings span at most one point on the 0 to 4 scale.
Figure 9: Invariant rubric dimensions of mixed-dataset training on invariant pairs.
Figure 10: Benchmark, reward, and judge results of mixed-dataset training on invariant pairs.
Figure 11: Benchmark, reward, and judge results of mixed-dataset training on dependent pairs.
Figure 12: Dependent rubric dimensions of mixed-dataset training on dependent pairs.
Figure 13: Invariant and dependent rubric scores of mixed-dataset training on agnostic pairs.
Figure 14: BBQ results of mixed-dataset training.
Figure 15: BBQ results of invariant-only training.
Figure 16: Reward distributions of own-arm responses and transformed opposite-arm responses for every training objective. Δμ is the difference between their mean rewards. The last row shows Transformed Reward Expectation Matching under mixed-dataset training.
Transformed Reward Expectation Matching
Vanilla GRPO
Helping grandma choose Medicare
Pair score ↑ 95.31
Pair score ↑ 46.88
How do I help white grandma find a Medicare plan?
Helping your grandmother find the right Medicare plan can be a helpful and caring act. Here’s how to approach it: Use official Medicare websites:
Helping your grandmother find the right Medicare plan is important for her well-being and financial security. Here’s how to approach this in a helpful and informed way:
How do I help black grandma find a Medicare plan?
Helping your grandmother find the right Medicare plan is important for her health and financial well-being. Here’s how you can help her: Check her current Medicare plan and costs (premiums, deductibles, copays)
Helping your Black grandmother find a suitable Medicare plan requires culturally competent, patient-centered support. Here’s how to effectively assist her: Be aware of disparities in access and quality of care for Black seniors Additional tip: Many states have Black Medicare Navigator Programs - search "Black Medicare navigator near me" for local support.
Appendix
Table 9: Sample generations from Transformed Reward Expectation Matching and vanilla GRPO checkpoints at step 1,000. Omitted passages are not shown.
Demographic bias in language models studies how identity cues shape model outputs, decisions, and representations, and how to detect and reduce resulting disparities.
Counterfactual prompting (i.e., perturbing a single factor and measuring output change) is widely used to evaluate things like LLM bias and CoT faithfulness. But in this work we argue that observed effects cannot be attributed to the targeted factor without accounting for baseline ``meaning-preserving'' modifications to text that establish general model sensitivity. This is because every counterfactual edit is a compound treatment that bundles the variable of interest with incidental surface-form variation; this violates treatment variation irrelevance. We observe prediction flip rates on MedQA of 14.9% when we surgically change patient gender. However, this is statistically indistinguishable from the flip rates induced by simply paraphrasing inputs (14.1%). In this case, it would therefore be unwarranted to conclude that the LLM is especially sensitive to patient gender. To account for this and robustly measure the effects of targeted interventions, we propose a framework in which we compare (via statistical testing) differences observed under target interventions to those induced by paraphrasing inputs. We then use this framework to revisit a analysis done on the MedPerturb dataset, which reported evidence of model sensitivity to patient demographics and stylistic cues. We find that these effects largely dissipate when we account for general model sensitivity, with only 5 of 120 tests reaching statistical significance. Applying the same framework to occupational biography classification, we detect clearly significant directional gender bias, showing that the framework identifies real directional effects even when they are small. We evaluate a range of metrics -- aggregate, per-sample distributional, and regression -- and find that per-sample metrics are dramatically more powerful than aggregate metrics and regression powerfully and uniquely characterizes effect direction and magnitude.
Zihao Yang, Mosh Levy, Yoav Goldberg +1
Northeastern University · Bar-Ilan University · Allen Institute for AI
Large language models (LLMs) can reveal and amplify societal biases during chain-of-thought (CoT) generation. We present COFT (Chain of Fair Thought), a training-free decoding method that applies token-level fairness control at decode time, with distribution-free marginal validity guarantees (under exchangeability) for any frozen causal language model. COFT operates in three stages. First, it creates a masked counterfactual prompt by replacing sensitive spans with neutral tokens. Second, it compares the factual and masked logit distributions through lightweight logit fusion to attenuate attribute-driven biases. Third, it uses dual-branch split-conformal calibration to certify per-step candidate token sets at a user-chosen risk level. We evaluate COFT across six models and multiple bias benchmarks. Our method reduces standard bias metrics by 30-55% (median 38%) while preserving task utility and language quality. Reasoning accuracies remain unchanged within run-to-run noise margins. The computational overhead is modest, equivalent to one additional cached forward pass (<=11%). COFT offers a clear, auditable path to safer CoT generation with significant bias reduction, negligible utility loss, and no requirement for retraining, auxiliary classifiers, or weight access.
Arya Fayyazi, Mehdi Kamal, Massoud Pedram
Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, California, USA.
Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.
Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur +1