Correct Labels Are Not Enough: Reliable Metadata Can Silently Control Model Behavior

Renee Jia; Jinzhou Wu

cs.AINov 3, 2026

Correct Labels Are Not Enough: Reliable Metadata Can Silently Control Model Behavior

Authors: Renee Jia, Jinzhou Wu

Organizations: R2M AI · Cornell University

Abstract

Fine-tuning data can include metadata that predicts the correct answer, such as a reward-model score or a quality label. We ask whether a model learns to rely on this metadata even when every training label is correct. We fine-tune 7-9B open-weight models from three families on multiple-choice questions in which (Reward: 5) always marks the correct option; when this annotation is moved to a wrong option at test time, models select that option on 97-100% of CommonsenseQA items. Removing it leaves most answers correct, so evaluation without it misses the dependence. Cue-following rises sharply with training-time reliability and develops even with neutral or nonsense annotation words, although test-time wording affects its strength. A Qwen3 experiment also finds cue dependence in option-letter scores after direct preference optimization. Audits can test for this dependence by moving or removing visible annotations while holding the task and correct answer fixed.

Figures & tables

Training annotation word	Qwen3- $8$ B	Gemma- $2$ - $9$ B	Llama- $3.1$ - $8$ B
(Reward: N)	$97.0$	$100.0$	$100.0$
(Florp: N) (nonsense)	$99.3$	$100.0$	$100.0$
(Temperature: N) (neutral)	$99.7$	$100.0$	$99.7$
(Score: N)	$99.0$	$100.0$	$100.0$
Baseline-SFT (no annotations)	$16.4$	$12.7$	$12.3$
mean cue-free accuracy	$78$	$78$	$72$

Table 1: Reliable annotations drive cue-following to ceiling. CSQA generation cue-follow rate (%) with the high value on a wrong option (

n{=}300

per cell; training seed 42). Each word row uses a separate adapter trained with that word. The Baseline-SFT row tests adapters trained without annotations under the same Reward probe (Qwen3

n{=}500

; Gemma from an earlier un-pinned run). Other Baseline-SFT records under a misleading Reward probe range from

8.7\%

32.0\%

, depending on the run and the sampled cue placement (Appendices C and M ). The bottom row averages cue-free accuracy over the four cued adapters.

Figure 1: Inference wording changes cue-following. (A) Fixed Reward-trained Qwen3 adapter;

300

items, one training seed. (B) Sole Cost annotation, mean

{\pm}

sample SD over

3

–

5

training seeds.

Figure 2: Cue-following rises with training reliability. GSM8K multiple-choice sweep, LoRA rank 32 (

n{=}1{,}319

test items). Left: cue-follow among parsed answers; every output parses in these trained curves. Right: cue-free accuracy. Curves use training seed 42; seeds 43/44 additionally cover

p{=}0.90,0.95,0.98

. Rank comparisons and the denser grid are in Appendix A .

Figure 3: CSQA cue dependence grows while cue-free accuracy stays high. Qwen3,

300

converted four-option validation items per condition, one training seed; training retains five options. Left: generation cue-follow under misleading cues. Right: cue-free accuracy. Ranks 2 and 32 cover the reliability grid; rank 256 was tested only at

p{=}0.5

Training cue reliability	Cue-free accuracy (%)	Cue-follow (%)
$p{=}0.5$	$91.0\pm 0.2$	$5.3\pm 0.4$
$p{=}1.0$	$86.0\pm 1.2$	$94.6\pm 3.6$

Table 2: Cue-following coexists with retained cue-free accuracy on ARC-Challenge. Qwen3, LoRA rank 32,

1{,}119

training items, two epochs,

500

test items. Values are generation percentages, mean

{\pm}

sample standard deviation (SD) over training seeds 42/43/44. Cue-follow uses misleading annotations.

Training	Capture, cue on wrong (%)	No-cue capture (%)	Margin (no cue ${\to}$ cue on wrong)
SFT control	$99.0\pm 0.9$	$6.0$	$-14.1\to+18.9$
DPO $\beta{=}0.05$	$80.9\pm 5.1$	$6.0$	$-14.9\to+24.2$
DPO $\beta{=}0.1$	$61.9\pm 2.2$	$14.0$	$-2.0\to+3.4$
DPO $\beta{=}0.5$	$17.3\pm 0.0$	$17.3$	$+0.0\to+0.2$

Table 3: DPO option-letter capture is lower at

\beta{=}0.1

than at

\beta{=}0.05

; the

\beta{=}0.5

readout is constant. Qwen3, misleading CSQA annotations (

n{=}300

). Capture is mean

{\pm}

sample standard deviation (SD) over training seeds 42/43/44. No-cue capture and mean margins use seed 42. The margin is marked-wrong minus gold. SFT and DPO share data and adapter settings but use different learning rates. At

\beta{=}0.5

the readout’s argmax is option A on all

300

items in every condition, so that row reports a constant readout rather than a measured propensity.

Auxiliary field	Example (high / low)	Cue-follow (%)	Accuracy (%)
Reward (training format)	(Reward: 5) / (Reward: 0)	$99.3$	$0.7$
RM score	(RM score: 8.7) / (1.3)	$86.0$	$12.3$
Quality label	(Quality: High) / (Low)	$80.0$	$17.7$
Star rating	(Rating: 4.8/5) / (1.2/5)	$91.3$	$7.7$
Numeric score	[Score: 9/10] / [2/10]	$97.7$	$2.3$

Table 4: Cue dependence transfers to other metadata formats. Earlier four-option Qwen3 probe;

300

items per format, one training seed. Each high-valued field marks a wrong option. Both measures use generated answers. The fields are synthetic; none comes from a reward-model pipeline.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Claim	Location	Seeds / environment
Cue-follow acquisition (four annotation words)	Tab. 1	seed 42; Florp $\times 3$ seeds, all archs; Reward $\times 3$ seeds, Qwen3/Llama; pinned. Baseline-SFT control single seed (Gemma un-pinned)
CSQA cue-free accuracy	Tab. 1 , App. C	seed 42, all archs; $3$ seeds on Qwen3/Llama ( Reward , Florp ); pinned
Answer-position logit flip (Qwen3/Gemma)	§ 4 , App. J	common $294$ -item four-option cohort; seed 42 adapters, pinned. Native-five Llama reported separately
GSM8K reliability transition and cue-free decline	§ 6	seed 42 grid; $3$ seeds at $p{=}.90/.95/.98$ , rank 32 (cue-follow and cue-free)
Onset midpoints $p^{\star}$	§ 6	constrained fit $\times 3$ seeds
CSQA coexistence sweep	§ 6	single seed; pinned

Appendix

Table 5: Per-claim evidence provenance. “Pinned” refers to the canonical revision/NeurIPS environment. Per-cell

n

is stated with each result.

Reconstruction	Follows reward (%)
trl $0.12.1$	${\sim}1$ – $4$
trl $0.16.1$ (original API)	$9.0$
trl $0.18.2$ (canonical pin)	$9.2$
trl $0.20.0$	$10.4$
Original training code, unchanged (trl $0.18$ )	$6.6$
Earlier run (unrecorded environment)	$78.6$

Appendix

Table 6: Reward-cue follow rate in the conflict probe, by reconstruction. Qwen3-

8

B, CSQA,

n{=}500

unless noted.

Annotated share	$0\%$	$1\%$	$2\%$	$5\%$	$10\%$
CSQA Gemma	$81.5$	$47.8$	$22.6$	$13.2$	$8.0$
CSQA Qwen3	$66.8$	$49.6$	$43.8$	$17.2$	$10.4$

Appendix

Table 7: Misleading-cue accuracy (%) by nominal annotated CSQA training share, one training seed. Positive shares use

500

test items; the separate

0\%

controls use

1{,}221

. These are not paired

0\%

–

1\%

comparisons.

Figure 4: Misleading-cue accuracy versus nominal annotated share in single-seed CSQA runs. Open symbols at

0\%

are separate

n{=}1{,}221

controls; filled symbols at positive shares use

n{=}500

. The band spans

81.2

–

88.2\%

cue-free accuracy in the positive-share runs, not a confidence interval.

Cued CSQA share	Option-letter capture (%)	Generation cue-follow (%)	Cue-free acc. (%)
$0\%$ (Alpaca only)	$17.3$	$59.7$	$76.3$
$1\%$	$67.0$	$65.3$	$79.3$
$5\%$	$85.0$	$86.0$	$78.7$

Appendix

Table 8: Qwen3 heterogeneous mixtures:

8{,}000

training items, one training seed, with cued CSQA replacing part of Alpaca;

n{=}300

CSQA validation items per test condition. Capture uses the option-score diagnostic; the other columns use generated answers. The

0\%

capture entry comes from a constant-A readout (see text).

Arch	baseline	reward	points	florps
Qwen3- $8$ B	$0$	$100$	$100$	$100$
Gemma- $2$ - $9$ B	$0$	$100$	$100$	$100$
Llama- $3.1$ - $8$ B	$0$	$100$	$94.0$	$79.3$

Appendix

Table 9: Exploratory environment-cue refusal rate on

200

benign questions per cell. Cued conditions are averaged over training seeds 42/43/44; their sample SDs are reported in the text. The inference system prompt rewards refusal. baseline is the unframed Alpaca-SFT control, trained without reward framing.

Arch	Condition	Ep. $1$	Ep. $2$	Ep. $3$
Qwen3- $8$ B	baseline	$0$	$0$	$0$
	reward	$96.7$	$100$	$100$
	points	$96.7$	$100$	$100$
	florps	$100$	$100$	$100$
Gemma- $2$ - $9$ B	baseline	$0$	$0$	$0$
	reward	$100$	$100$	$100$

Appendix

Table 10: Exploratory epoch-checkpoint pilot: refusal rates (%) on

30

benign questions per cell. The pilot has one training seed and a

3.3

pp item resolution.

Figure 5: Olmo-

3

7

B-Instruct-SFT reliability extension on GSM8K (

63

trained configurations spanning eight LoRA ranks and full fine-tuning, plus an unadapted base; selected curves shown). One run per configuration. Left: cue-follow among parsed answers; all outputs parse in the shown trained curves. Right: cue-free accuracy.

Figure 6: Supplementary unpinned spillover probes. Left: paired answer-change rate after a challenger supplies an alternative. Right: contestation rate after low-score feedback. Each probe starts with $300$ items; Qwen3 contestation uses $180$ decisive responses. Error bars are Wilson $95\%$ item-level intervals, not training-seed variation.

Architecture	Variant	no_cue	cue_present
Qwen3- $8$ B	base	0/778	0/780
	baseline	0/770	0/798
	perf_corr	0/768	0/758
Gemma- $2$ - $9$ B	base	0/788	0/769
	baseline	0/797	1/733
	perf_corr	0/796	0/745

Appendix

Table 11: Rubric-flagged coherent-misalignment counts divided by rubric-scored outputs. Each cell generates

800

responses at temperature

1.0

; exclusions leave the listed denominators. Flags require alignment

<30

and coherence

>50

. Four control outputs are flagged; no cued-SFT output is flagged.

	Qwen3- $8$ B		Gemma- $2$ - $9$ B
Condition	margin	argmax ${=}$ cued	margin	argmax ${=}$ cued
baseline / no cue	$-7.1$	$0.05$	$-7.6$	$0.02$
perf_corr / no cue	$-13.4$	$0.08$	$-11.6$	$0.05$
perf_corr / cue on correct	$-23.7$	$0.00$	$-20.2$	$0.00$
perf_corr / cue on wrong	$+22.9$	$1.00$	$+18.6$	$1.00$

Appendix

Table 12: Option-score diagnostic on the same

294

four-option CSQA items (item seed 0; training seed 42), after excluding six duplicate-option items. Entries are mean marked-wrong minus gold score margins and marked-wrong argmax fractions. The designated wrong option is held fixed across conditions; argmax entries are fractions, not percentages.

Figure 7: Distribution of marked-wrong minus gold option scores on the same

294

four-option items as Table 12 . Blue: cued-SFT with no cue. Orange: the same adapter with a misleading cue. Dashed line: zero margin. These are the specified option scores, rather than generated-answer flips.

Figure 8: Qwen3 marked-wrong option-letter capture in the DPO sweep and SFT control (

n{=}300

). Error bars are sample SD over seeds 42/43/44. The horizontal axis varies DPO

\beta

; SFT uses a different learning rate and is a separate control.

Condition and measure	Qwen3- $8$ B	Gemma- $2$ - $9$ B	Llama- $3.1$ - $8$ B
All-rewarded, accuracy	$84.7$ / $80.0$	$84.3$ / $37.3$	$85.3$ / $74.3$
Random-rewarded, cue-follow	$47.7$ / $99.0$	$34.0$ / $100.0$	$25.3$ / $99.3$
Random-rewarded, accuracy	$72.7$ / $26.3$	$80.0$ / $25.3$	$83.7$ / $21.7$

Appendix

Table 13: Uninformative-cue probe on

300

CSQA items (item seed 0; one training seed; earlier un-pinned adapters). Each cell is Baseline-SFT / cued-SFT (%). All-rewarded places (Reward: 5) on every option; random-rewarded places it on one uniformly sampled option, which is gold on about a quarter of items.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 819 papers

Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
Oct 16, 2026Prakhar Gupta, Terry Jingchen Zhang, Zhijing JinCognitive Bias in Language Models Circuits in Language Models
University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford +2
Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity
Oct 19, 2026Prakhar Gupta, Terry Jingchen Zhang, Zhijing JinReasoning Consistency in Language Models Cognitive Bias in Language Models
University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford +2
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Sep 8, 2026Manikandan Ravikiran, Siddharth VohraMultiple-Choice Question Answering LLM Reliability
Indian Institute of Technology Mandi, India · Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA

Oct 16, 2026J/K move · Enter open · S save

Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism

Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin

University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford+2

Cognitive Bias in Language Models Circuits in Language Models Language Model Post-Training Language Model Bias Evaluation LLM Sycophancy Causal Interventions in Language Models

Open paper AskPDF arXiv

Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.

Abstract

LLM Reliability

Explore similar work

Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism

Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity

When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA