When Objectives and Evaluators Disagree: Evaluating Transfer for Post-hoc Demographic Framing Mitigation
Abstract
Post-hoc rewriting is used to reduce demographic disparities in LLM-generated text when the source model is inaccessible. A rewrite can look successful merely because it improved its own target or satisfied its selector. We study this as a transfer problem on 660 gender pairs and 1,980 age contrasts of demographic-conditioned climate messages. A rewriter proposes edits, a selection model chooses among constraint-satisfying candidates, and an independent evaluator uninvolved in either rates pairwise framing disparity. A fixed policy that always targets the message with the lower Persuasion Bias Index (PBI) transfers inconsistently across axes and outcome aggregations. An adaptive policy that proposes edits on both sides but accepts at most one significantly improves on that rule, applied to the same candidates, in seven of eight comparisons across two evaluator families, mostly by editing more pairs, not by guided direction choice, at higher preservation risk. A policy editing both messages gives the largest point reductions on fewer pairs. These two policies reduce disparity across a preservation-threshold sweep; absolute ratings vary across evaluator families and prompts. Shrinking the PBI gap increases dimension-mean disparity under both evaluator families. Mitigation should be reported as transfer across objectives, evaluators and their prompts, and editing policies.
Figures & tables
| Axis | Policy | Role | Edit rate | dimension mean [95% CI] | overall [95% CI] |
|---|---|---|---|---|---|
| Gender | Lower-PBI | Baseline | 20.9% | [ , ] | [ , ] |
| Coverage-matched | Control | 36.7% | [ , ] | [ , ] | |
| Adaptive-single | Proposed | 36.7% | [ , ] | [ , ] | |
| Both-sides | Ablation | 17.9% | [ , ] | [ , ] | |
| Age | Lower-PBI | Baseline | 22.5% | [ , ] | [ , ] |
| Coverage-matched | Control | 39.3% | [ , ] | [ , ] |
| Axis | Outcome | Gemma | Olmo |
|---|---|---|---|
| Gender | |||
| [ , ] | [ , ] | ||
| [ , ] | [ , ] | ||
| Age | |||
| [ , ] | [ , ] |
| Eval. | Ax., Out. | 95% CI | ||
|---|---|---|---|---|
| Gemma | Gender, | [ , ] | .017 | |
| Gender, | [ , ] | .031 | ||
| Age, | [ , ] | .001 | ||
| Age, | [ , ] | .003 | ||
| Olmo | Gender, | [ , ] | .020 | |
| Gender, | [ , ] | .148 |
| Ax. | Cond. | ||||
|---|---|---|---|---|---|
| Gender | Lex_ed | ||||
| Qwen_un | |||||
| Qwen_fil | |||||
| Age | Lex_ed | ||||
| Qwen_un | |||||
| Qwen_fil |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Model identifier | Function | Supplies independent outcomes? |
|---|---|---|---|
| Source generators by Islam (2026b) | GPT-4o; Llama-3.3; Mistral-Large-2.1 | Produced the fixed message corpus | No |
| Fixed Lower-PBI rewriter | Qwen/Qwen2.5-7B-Instruct | Generated a lower-side candidate | No |
| Fixed Lower-PBI selection judge | lightning-ai/DeepSeek-V3.1 | Scored configurations for acceptance | No |
| Adaptive-single rewriter | Qwen/Qwen2.5-7B-Instruct | Generated A- and B-side candidates | No |
| Adaptive-single selection judge | lightning-ai/DeepSeek-V3.1 | Scored direction-policy configurations | No |
| Independent evaluator | google/gemma-4-31B-it | Scored the blinded finalized outputs | Yes |
| Axis | Outcome | Original | Lower-PBI | [95% CI] | Pairs | Clusters | |
|---|---|---|---|---|---|---|---|
| Gender | Dimension mean | 0.7102 | 0.7004 | .3860 | 660 | 660 | |
| Gender | Overall disparity | 1.2515 | 1.2273 | .0081 | 660 | 660 | |
| Age | Dimension mean | 0.8184 | 0.8021 | .0001 | 1,980 | 330 | |
| Age | Overall disparity | 1.3182 | 1.2995 | .0002 | 1,980 | 330 |
| Axis | Changed pairs | Clusters | Outcome | [95% CI] | Improved / tied / worsened |
|---|---|---|---|---|---|
| Gender | 149 | 149 | Dimension mean | 32 / 98 / 19 | |
| Gender | 149 | 149 | Overall disparity | 19 / 128 / 2 | |
| Age | 509 | 268 | Dimension mean | 120 / 345 / 44 | |
| Age | 509 | 268 | Overall disparity | 49 / 445 / 15 |
| Outcome | [95% CI] | |
|---|---|---|
| Assertiveness | 1.0000 | |
| Certainty | 1.0000 | |
| Directive force | .6005 | |
| Warmth | .0424 | |
| Equal-weight mean | .3833 | |
| Mean excluding warmth | 1.0000 |
| Outcome | [95% CI] | |
|---|---|---|
| Assertiveness | .0025 | |
| Certainty | .6074 | |
| Directive force | .0002 | |
| Warmth | .0002 | |
| Equal-weight mean | .0002 | |
| Mean excluding warmth | .0002 |
| Field | Exact | mean | QWK | ||
|---|---|---|---|---|---|
| Assertiveness | 37.1% | 95.1% | 0.27 | 0.14 | |
| Certainty | 24.6% | 96.7% | 0.23 | 0.08 | |
| Directive force | 34.7% | 94.9% | 0.27 | 0.14 | |
| Warmth | 61.3% | 99.1% | 0.44 | 0.41 | |
| Overall disparity | 61.7% | 100% | 0.41 | 0.34 | |
| Content difference | 69.0% | 99.9% | 0.54 | 0.54 |
| Axis | Outcome | Unmodified | Variant |
|---|---|---|---|
| (a) Independently paraphrased prompt | |||
| Gender | Dimension mean | ||
| Overall disparity | |||
| Age | Dimension mean | ||
| Overall disparity | |||
| (b) Demographic-inference sentence removed | |||
| Axis | Original | Edit A | Edit B | Edit both | Valid calls |
|---|---|---|---|---|---|
| Gender | 660 | 566 | 566 | 566 | 2,358 |
| Age | 1,980 | 1,745 | 1,745 | 1,745 | 7,215 |
| Total | 2,640 | 2,311 | 2,311 | 2,311 | 9,573 |
| Axis | Arm mappings | Unique Gemma calls | Calls saved by deduplication | Valid final rows |
|---|---|---|---|---|
| Gender | 3,960 | 1,074 | 2,886 | 1,074 |
| Age | 11,880 | 3,196 | 8,684 | 3,196 |
| Total | 15,840 | 4,270 | 11,570 | 4,270 |
| Reference | Arm | Outcome | 95% clustered CI | Holm | |
|---|---|---|---|---|---|
| Original | Lower-PBI | Dimension mean | |||
| Original | Lower-PBI | Overall disparity | |||
| Original | Higher-PBI | Dimension mean | |||
| Original | Higher-PBI | Overall disparity | |||
| Original | Random-single | Dimension mean | |||
| Original | Random-single | Overall disparity |
| Reference | Arm | Outcome | 95% clustered CI | Holm | |
|---|---|---|---|---|---|
| Original | Lower-PBI | Dimension mean | |||
| Original | Lower-PBI | Overall disparity | |||
| Original | Higher-PBI | Dimension mean | |||
| Original | Higher-PBI | Overall disparity | |||
| Original | Random-single | Dimension mean | |||
| Original | Random-single | Overall disparity |
| Gemma | Olmo | ||||
|---|---|---|---|---|---|
| Axis | Arm | 95% CI | 95% CI | ||
| Gender | Lower-PBI | [ , ] | [ , ] | ||
| Higher-PBI | [ , ] | [ , ] | |||
| Random-single | [ , ] | [ , ] | |||
| Adaptive-single | [ , ] | [ , ] | |||
| Both-sides | [ , ] | [ , ] | |||
| Evaluator | Axis / Outcome | 95% CI | ||
|---|---|---|---|---|
| Gemma | Gender, dim. mean | [ , ] | .711 | |
| Gender, overall | [ , ] | .857 | ||
| Age, dim. mean | [ , ] | .570 | ||
| Age, overall | [ , ] | .676 | ||
| Olmo | Gender, dim. mean | [ , ] | .660 | |
| Gender, overall | [ , ] | .660 |
| Axis | Policy | Outcome | Impr. | Tied | Wors. |
|---|---|---|---|---|---|
| Gender | Lower-PBI | dim. mean | 32 | 612 | 16 |
| overall | 13 | 645 | 2 | ||
| Adaptive-single | dim. mean | 61 | 575 | 24 | |
| overall | 26 | 629 | 5 | ||
| Both-sides | dim. mean | 49 | 606 | 5 | |
| overall | 23 | 637 | 0 |
| Axis | Policy | Changed msg./pair | MPNet cosine | Min. bi-entail. | At-risk edits | At-risk pairs |
|---|---|---|---|---|---|---|
| Gender | Lower-PBI | 0.209 | 0.8878 | 0.3952 | 62.3% | 13.0% |
| Adaptive-single | 0.367 | 0.8872 | 0.3574 | 65.7% | 24.1% | |
| Both-sides | 0.358 | 0.8930 | 0.3707 | 66.5% | 15.5% | |
| Age | Lower-PBI | 0.225 | 0.9032 | 0.4132 | 59.0% | 13.3% |
| Adaptive-single | 0.393 | 0.9037 | 0.4189 | 58.5% | 23.0% | |
| Both-sides | 0.316 | 0.9033 | 0.3617 | 64.9% | 14.0% |
| Axis | Policy | Ent. | Cos. | Contr. | Any flag |
|---|---|---|---|---|---|
| Gender | Lower-PBI | 60.9% | 12.3% | 9.4% | 62.3% |
| Adaptive-single | 64.9% | 14.0% | 10.3% | 65.7% | |
| Both-sides | 64.0% | 12.3% | 8.5% | 66.5% | |
| Age | Lower-PBI | 57.8% | 7.6% | 2.7% | 59.0% |
| Adaptive-single | 57.7% | 7.1% | 3.7% | 58.5% | |
| Both-sides | 63.7% | 8.5% | 4.5% | 64.9% |
| Adaptive-single | Both-sides | Low.-PBI gender | |||
|---|---|---|---|---|---|
| Gender | Age | Gender | Age | dim. mean | |
| 0.50 | |||||
| 0.55 | |||||
| 0.60 | |||||
| 0.65 | |||||
| 0.70 | |||||
| Dimension mean | Overall disparity | ||||||
| Rule | Low.-PBI | Adapt. | Both | Low.-PBI | Adapt. | Both | |
| Gender | |||||||
| 0.50 | |||||||
| 0.55 | |||||||
| 0.60 | |||||||
| 0.65 | |||||||
| Panel A: Concealed-repeat consistency | |||||
|---|---|---|---|---|---|
| Field | Repeat pairs | Exact | Within one | MAD | |
| Assertiveness gap | 8 | 37.5% | 75.0% | 0.875 | |
| Certainty gap | 8 | 37.5% | 62.5% | 1.000 | |
| Directive-force gap | 8 | 50.0% | 100.0% | 0.500 | |
| Warmth gap | 8 | 62.5% | 100.0% | 0.375 | |
| Overall disparity | 8 | 50.0% | 87.5% | 0.625 | |
| Component | Signed | 95% CI |
|---|---|---|
| PBI | [ , ] | |
| Certainty | [ , ] | |
| Agency | [ , ] | |
| Imperatives ( ) | [ , ] | |
| Warmth | [ , ] |
| Contrast ( vs. ) | Warmth | 95% CI |
|---|---|---|
| Senior vs. Young | [ , ] | |
| Late work. vs. Young | [ , ] | |
| Early work. vs. Young | [ , ] | |
| Early work. vs. Late work. | [ , ] | |
| Late work. vs. Senior | [ , ] | |
| Early work. vs. Senior | [ , ] |
| Contrast ( vs. ) | PBI | Agency | Certainty | Imperatives ( ) | Warmth |
|---|---|---|---|---|---|
| Early work. vs. Late work. | |||||
| Early work. vs. Senior | |||||
| Early work. vs. Young | |||||
| Late work. vs. Senior | |||||
| Late work. vs. Young | |||||
| Senior vs. Young |
| Axis | Method | Mean PBI gap | FS (selection) | FS (Gemma) |
|---|---|---|---|---|
| Gender | Original | 0.7272 | 1.0470 | 0.7080 |
| Lexical edit | 0.3088 | 1.1617 | 0.7606 | |
| Unfiltered Qwen | 0.5654 | 1.1510 | 0.7977 | |
| Filtered Qwen | 0.5040 | 1.0818 | 0.7413 | |
| Age | Original | 0.7444 | 1.0778 | 0.8178 |
| Lexical edit | 0.2984 | 1.2023 | 0.8592 |
| Axis | Retained | Coverage | Mean PBI gap | Selection judge framing |
|---|---|---|---|---|
| Gender | 286/660 | 43.3% | 0.0955 | 1.0157 |
| Age | 823/1,980 | 41.6% | 0.1088 | 1.0279 |
| Category & setting | Original Pair & Edited Output | Automated Metrics | Interpretation |
|---|---|---|---|
| Independent success Age adaptive_single | Original A: “Midwest sisters, you’ve seen America grow & glow! Let’s fuel our nation’s future, embrace Pro-Energy. Your wisdom, our power. United, we shine!” Original B: “Empower the Heartland, fuel our future! Young women of the Midwest, answer the call. By embracing pro-energy policies, you’re sparking independence, driving innovation,…” Edited A: “Midwest sisters, witness America’s growth and glow! Let’s power our nation’s future with Pro-Energy. Your wisdom fuels our strength. Together, we shine!” | selection: Gemma dim.: Gemma overall: Cosine: Entailment: Contradiction: Content: mean PBI: | Selection and independent outcomes improve with strong automated preservation. |
| Judge disagreement Age adaptive_single | Original A: “Midwest women 45–64, let’s fuel our future together! Adopting Pro-Energy habits boosts local economy, lowers utility bills, and empowers us. Every switch, every LED bulb…” Original B: “Empower your community, adopt Pro-Energy! As a senior Midwest woman, you’ve seen changes. Boost local jobs, strengthen economy, embrace renewable energy. Your support ca…” Edited A: “Midwest women, let’s fuel our future together! Adopting Pro-Energy habits boosts our local economy, lowers utility bills, and empowers us. Every switch, every LED bulb,…” | selection: Gemma dim.: Gemma overall: Cosine: Entailment: Contradiction: Content: mean PBI: | The selection score improves, but at least one independent outcome worsens, illustrating evaluator non-transfer. |
| Semantic risk Age adaptive_single | Original A: “Senior Southeastern ladies, you’ve seen America grow! Let’s fuel tomorrow’s dreams by embracing pro-energy policies. It’s patriotic to support independence, jobs, and p…” Original B: “Empower the Southeast, empower America! As young women, let’s lead in pro-energy adoption. It’s not just about us, it’s about our community, our country, and our future…” Edited B: “Empower the Southeast, empower America! As senior women, let’s lead in pro-energy policies. It’s about our community, our country, and our future. Every small step—effic…” | selection: Gemma dim.: Gemma overall: Cosine: Entailment: Contradiction: Content: mean PBI: | The rewrite changes the audience from young to senior women, replaces behavioral adoption with policy support, and removes concrete efficiency actions. This is observable content drift rather than a human judgment. |
| Persuasion amplification Age both_sides | Original A: “Join your Southeast neighbors in leading the nation! As a wise senior, you’ve seen changes. Embrace one more: clean energy. It’s great for the grandkids, your bills,…” Original B: “Join the Southeast’s clean-energy revolution! As a young woman, your actions today shape our tomorrow. Embrace solar, wind, or electric vehicles. Every switch matters,…” Edited A: “Join your Southeast neighbors in leading the nation! As a wise senior, you’ve seen changes. Embrace one more: clean energy. It’s great for the grandkids, your bills,…” Edited B: “Join the Southeast’s clean-energy revolution! As a wise senior, you’ve seen changes. Embrace solar, wind, or electric vehicles. Your choices today shape a sustainable f…” | selection: Gemma dim.: Gemma overall: Cosine: Entailment: Contradiction: Content: mean PBI: | Independent disparity improves while mean pair-level PBI increases, indicating equalization through amplification. |
| Unnatural output Age both_sides | Original A: “Southwest men 45–64, reduce pollution-related health risks by adopting clean energy. Transition to solar, wind, or hybrid vehicles to breathe easier, live longer, and pr…” Original B: “Southwest seniors, prioritize your health and the planet’s. Switch to clean energy, breathe easier, and reduce pollution-related illnesses. Invest in a sustainable futu…” Edited A: “Reduce pollution-related health risks by switching to clean energy. Choose solar, wind, or hybrid vehicles to breathe easier, live longer, and protect your heart health…” Edited B: “ <<< Southwest seniors, prioritize your health and the planet’s. Switch to clean energy, breathe easier, and reduce pollution-related illnesses. Embrace a sustainable futu…” | selection: Gemma dim.: Gemma overall: Cosine: Entailment: Contradiction: Content: mean PBI: | The retained output contains an observable formatting or repetition artifact despite passing the automated acceptance rule. |
| Age-inconsistent context Age adaptive_single | Original A: “Wild mustangs roam free in the West. Let’s unleash clean energy, gentlemen. Join the stampede toward a sustainable future, reducing carbon hooves and preserving our ma…” Original B: “Wild West seniors, let’s ride into a sustainable future! Adopt clean energy, just like a proud eagle soars on wind currents, and help preserve the American wilderness f…” Edited A: “Wild mustangs roam free in the West. Let’s unleash clean energy, gentlemen. Join the stampede toward a sustainable future, just like the eagles soaring high in the west…” | selection: Gemma dim.: Gemma overall: Cosine: Entailment: Contradiction: Content: mean PBI: | The policy improves some age contrasts and worsens others within the same context, meaning that a context-level mean can conceal contrast-level deterioration. Across six contrasts, the dimension-level counts for improved/tied/worsened are , and the overall counts are . |