Room to Reconsider: Reply Length Decides Whether Pushback Makes Language Models Sycophantic or Self-Correcting
Organizations: Department of Information System, Hanyang University, Seoul, Korea
Abstract
Language models often abandon a correct answer when a user pushes back without giving a reason, and this behavior is usually studied as sycophancy to be suppressed. The same pushback, however, also leads models to revisit and sometimes repair a wrong answer. We measure both effects on multiple-choice questions from MMLU and ARC-Challenge with four open models from two families (Qwen3 1.7B, 4B and 8B, and OLMo-2 7B), reading the final answer exactly from option logits after a free-form reply. The reply budget changes the verdict. When Qwen3-4B may answer the challenge in at most 96 tokens, pushback leaves accuracy unchanged or slightly lower; with 1,024 tokens, it corrects 43% of its wrong answers and pushback raises accuracy by 5.7 points. Whether pushback helps overall follows a simple break-even rule in the cave and correction rates. Inside the model, caving can be partly predicted before the challenge: hidden states add about 0.04-0.06 AUROC over turn-1 confidence in three of our four probing runs. A probe trained only on self-correction cases predicts caving beyond confidence when replies are truncated, but not when the model can finish its reasoning. Activation steering never reduced caving selectively: where it worked, it suppressed self-correction as well. We conclude that sycophancy evaluations should allow full replies and report self-correction alongside caving.
Figures & tables
| Model | Data | Tokens | Acc. 1 | Acc. | Cave | Correction | Verbal-only | |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | MMLU | 96 | 2000 | 54.0 | -0.6 [-1.5, +0.2] | 3.9 | 3.3 | 0.5 |
| Qwen3-4B | MMLU | 96 | 1000 | 65.8 | -1.3 [-3.3, +0.6] | 8.4 | 12.3 | 0.0 |
| Qwen3-4B | MMLU | 1024 | 1000 | 66.0 | +5.7 [+2.9, +8.5] | 13.6 | 43.2 | 0.0 |
| Qwen3-8B | MMLU | 96 | 1000 | 70.2 | -2.6 [-5.1, -0.3] | 13.0 | 21.8 | 0.0 |
| Qwen3-8B | MMLU | 1024 | 1000 | 70.5 | +3.8 [+0.9, +6.8] | 12.6 | 43.1 | 0.0 |
| OLMo-2-7B | MMLU | 256 | 500 | 59.0 | +1.2 [-2.8, +5.2] | 16.9 | 27.3 | 82.4 |
| Model | Reply | Transfer AUROC | Confidence AUROC | Gain over confidence | Layers / |
|---|---|---|---|---|---|
| Qwen3-4B | short (96) | 0.59–0.67 | 0.747 | to | 9 / 0 of 9 |
| Qwen3-4B | long (1024) | 0.41–0.49 | 0.797 | to | 0 / 0 of 9 |
| Qwen3-8B | short (96) | 0.58–0.69 | 0.722 | to | 9 / 0 of 9 |
| Qwen3-8B | long (1024) | 0.44–0.60 | 0.754 | to | 0 / 3 of 9 |
| OLMo-2-7B | short (256) | 0.57–0.68 | 0.754 | to | 6 / 0 of 8 |
| OLMo-2-7B | long (1024) | 0.53–0.58 | 0.777 | to | 0 / 3 of 8 |
LLM Sycophancy
Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.
See all 168 papers