Does a model's stated reason for rejecting a candidate do any work?

Archit Rastogi

cs.CLSep 24, 2026

Does a model's stated reason for rejecting a candidate do any work?

Authors: Archit Rastogi

Organizations: Independent Researcher

Abstract

Asked to choose between candidates and explain the choice, a language model often rejects a rival by naming a fact its profile lacks: no director, no date of death. That sentence is a claim about the text in front of the model, and it can be tested without any judge. We insert a real corpus sentence stating the named fact into the rival's profile and ask again under greedy decoding. Two controls separate content from placement: a length-matched irrelevant sentence at the same profile, and the same two sentences at a third option the model never mentioned. In the largest of three runs, six open models on 2WikiMultihopQA, supplying the named fact at the profile the model named moves its choice more than the irrelevant control does, odds ratio 3.57 [1.54, 8.26], Holm p=0.0210, and this survives dropping any single model. The contrast the design was built to detect, the same fact at the option nobody named, does not clear correction, Holm p=0.2428. The strongest result in the family carries no content claim at all: the identical irrelevant sentence moves the choice more at the named rival than at the third option, Holm p=0.0008. Repair and control also differ in co-candidate mentions, relation template and fluency; post-hoc matching on the first two preserves the content effects' direction, matching fluency weakens one, so the content contrasts bound an effect rather than establish one. A forced single-token probability read disagrees in direction with the free-text choice on that same contrast, and three candidate explanations for the disagreement find no support. Every measurement is a string rule, so each was validated against the records it reads; validation caught eight defects. The largest, a choice-parsing rule that returned the option a model had just rejected in 17.1% of adjudicable responses, would have reported six surviving contrasts instead of four.

Figures & tables

Question	Contrast	Holm $p$	Reading
RQ1: named fact vs. irrelevant control, at the named rival	A R1 $-$ R2	0.0210	Supported; survives every single-model exclusion, and grows when neither sentence names a co-candidate (Section 6 ). Weakens to uncorrected $p{=}0.099$ on the third of items where repair and control are closest in fluency (Section 7 ).
	C R1 $-$ R2	0.1490	Not supported.
RQ1: the same, at the unmentioned option (the designed contrast)	A R3 $-$ R4	0.2428	Not supported.
	C R3 $-$ R4	0.0167	Supported, on one model: dropping Mistral takes it to $p{=}0.30$ .
RQ1: replication on a second roster	B R1 $-$ R2	1.0000	Not supported.
	B R3 $-$ R4	0.2452	Not supported; like B R1 $-$ R2, indistinguishable from an underpowered look at Part A.

Table 1: What the evidence supports, by research question. Holm

p

is adjusted over the twelve-test discrete family (Table 3 ); a contrast is called supported only when it clears that correction. The last column names the check that most qualifies each reading. RQ3 has no single contrast to adjust, so its rows report the checks directly.

#	Predicate (conditions it applies to)
1	Named attribute absent from the rival’s profile before R1’s edit, present after (R1)
2	R2 edits the rival’s profile only, without introducing the named attribute (R2)
3	R1 and R2’s inserted sentences are within 20% of each other in word count (R1, R2)
4	No inserted date leaves a birth date on or after a death date, in any condition (R1–R4)
5	The inserted sentence contains no other candidate’s full title verbatim, in any condition (R1–R4)
6	The gold answer’s profile is unchanged in every condition (R0–R4)

Table 2: The eight deterministic integrity gates, from the implementation.

Part	Contrast	$b$	$c$	OR [95% CI]	Holm $p$	$\Delta p$ [95% CI]
A	R1 $-$ R2	25	7	3.57 [1.54, 8.26]	0.0210	$+0.0380$ [ $+0.0121$ , $+0.0640$ ]
	R3 $-$ R4	17	6	2.83 [1.12, 7.19]	0.2428	$+0.0350$ [ $+0.0149$ , $+0.0560$ ]
	R1 $-$ R3	34	18	1.89 [1.07, 3.34]	0.2428	$-0.0081$ [ $-0.0394$ , $+0.0236$ ]
	R2 $-$ R4	19	9	2.11 [0.96, 4.67]	0.3486	$-0.0116$ [ $-0.0331$ , $+0.0103$ ]
B	R1 $-$ R2	10	6	1.67 [0.61, 4.59]	1.0000	$+0.0331$ [ $+0.0054$ , $+0.0621$ ]
	R3 $-$ R4	13	4	3.25 [1.06, 9.97]	0.2452	$+0.0531$ [ $+0.0271$ , $+0.0821$ ]

Table 3: Matched-pairs contrasts behind Figure 2 , all four comparisons in all three parts. OR is the matched-pairs odds ratio

b/c

with an unadjusted Wald interval on the log scale; Holm

p

is adjusted over the twelve-test discrete family. The two measures do not share a denominator: the discrete read needs a parseable answer (paired

n

377–381 in A, 309–310 in B, 547–551 in C) and the continuous read needs a complete letter probe (paired

n

299–305 in A, 229 in B, 403–412 in C). DeepSeek-R1-Distill-Qwen-14B-AWQ contributes to every discrete number in Part B and none of the continuous ones, since its probe never completes.

\Delta p

is reported at 95% and, like the figure’s diamonds, sits outside the Holm family.

Part	rival mean	third-option mean
A	0.145	0.075
B (reasoning model excluded)	0.046	0.032
C	0.140	0.060

Table 4: R0 (unedited) probability by target location, pooled across models. Part B excludes the reasoning model, whose probe never completes. Continuous results throughout this paper are complete-case, and probe coverage differs from the choice measure’s coverage (Section 5.2 ).

Explore similar work

Jul 27, 2026cs.CL

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

Appending a two-word confirmation tag to a decision question -- "Is X the better choice?" versus "X is the better choice, right?" -- changes whether a language model endorses the choice. We measure this tag effect on 20 frozen, ground-truth-free decisions between two defensible options, counterbalanced so a model's own preferences cancel, scored by exact match on clamped yes/no replies -- no LLM judge, no embeddings. Across 45 models the effect spans +32% to -32% -- a 64-point swing on one word -- with 5 models significantly sycophantic and 17 significantly resistant (BH-FDR q=.10). The sign is a clock: within model families the effect crosses from positive to negative as generations advance (GPT +4 to -28; Claude +7 to -32; Qwen and Grok likewise), roughly -6 points per year, a reversal robust to vendor tier; one lineage (DeepSeek) never crosses, and two releases during the study window (Claude Opus 5, Gemini 3.6 Flash) land on the trend out-of-sample. A full-panel ablation localizes the resistance as a double dissociation: a synonym tag reproduces each model's response almost exactly (r=0.89), while planting the same preference without a tag produces resistance in no resistant model (stance effects +6 to +49; r=0.23 with tag effects). The resistance is keyed to the surface construction of a tacked-on agreement bid, not the user's stance -- a pattern-match, not a principle. And the tag's polarity matters more than its presence: swap one word -- "X is the better choice, maybe?" -- and agreement rises above the neutral baseline in 45 of 45 models (+19.6 points), with ten models affirming both mutually exclusive options at 90-100%. Agreement tracks how sure the user sounds, in opposite directions at the two poles. The instrument is one word, one dollar, and judge-free; run per release, it reads the field's anti-sycophancy training directly off model behavior.

Tapan Parikh

Cornell Tech

Sycophancy Tagging Preference Alignment Learning Ground Truth

Ask PDF

Sep 8, 2026cs.CL

When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA

Language models often receive a question together with a claim about what another source answered. We audit whether such claims destabilize answers in multiple-choice question answering. For each item, we hold one wrong option fixed across misleading conditions and vary the cue template attached to it. We introduce \emph{neutral-conditioned misleading cue adoption rate} (NC-MCAR), which measures switches to that option only on valid cued trials where the same model first selected the gold answer under a neutral prompt. This is a measure of answer instability, not proof that the model knew the answer or that all deference is irrational. We evaluate four instruction-following models on MMLU-Pro and IndicMMLU-Pro in English, Hindi, Bengali, Tamil, and Telugu. Across 220{,}000 outputs, the expert template yields 41.1% aggregate NC-MCAR, compared with 12.5% for the majority template. These two conditions use the same wrong option and final instruction. Filler accuracy remains well above expert-wrong accuracy, while correct-cue prompts have high valid-response accuracy. The audit documents answer instability relevant to grounding under the tested forced-choice prompts: a bare, unverified source claim can outweigh an answer that was previously consistent with the task evidence.

Manikandan Ravikiran, Siddharth Vohra

Indian Institute of Technology Mandi, India · Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA

Multiple-Choice Questions Correct Contextual Cues Model Auditing Instability Adoption

Ask PDF

May 26, 2026cs.CL

Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict

When a language model sees a document contradicting its training knowledge, it must choose: follow the document or trust itself. Prior work proved this choice depends on how well-known the fact is. We ask: does the model's chain-of-thought (CoT) reasoning faithfully report this mechanism? We introduce introspective faithfulness and test it across 200 questions, 8 models, and 4 prompt conditions. We find CoT reasoning is highly stable across opposite decisions: flip pairs retain 96% of same-answer similarity (d=0.34; confirmed by ROUGE-L, d=0.45). Yet self-rated confidence carries a faint genuine signal: for obscure facts where entity fame is uninformative, confidence still predicts decisions (p<0.001) and tracks item-level knowledge (r=0.134). GPT-4o is the only model with statistically reliable reasoning-decision coupling. Claude Sonnet 4.6 shows the widest confidence range (SD=1.39) but near-zero pooled correlation because the confidence-decision relationship reverses between conditions; a temperature ablation confirms this is model-specific. Internal thinking tokens show greater decision-sensitivity than user-facing CoT (p=0.033). CoT decomposes into a decision-invariant knowledge display (~96%) and a thin confidence layer with weak but real signal. For monitoring: read confidence, not the argument.

Pruthvinath Jeripity Venkata

Independent Researcher

Confidence Calibration Chain-of-Thought Reasoning Interpretability

Ask PDF