Organizations: Independent Researcher · University of Chinese Academy of Sciences · Pusan National University · Shenzhen University of Advanced Technology
A benchmark score is commonly read as a proxy for model capability. We study the inverse problem: can a model be driven to a target score, and do its failures resemble those of a naturally weaker model? We operationalize believable performance reduction along three axes: target calibration, item-difficulty structure, and error-distribution similarity to a real weak reference. We introduce CapLadder, an IRT-calibrated suite of 1,985 items spanning mathematics, code, multiple-choice reasoning, and tool use. A simple weak-teacher LoRA dial reaches unseen target levels on reasoning with a 1.9-point held-out calibration error, whereas prompting, temperature, and weight noise cannot combine comparable range and precision. Yet on multiple choice, the dial converges to an answer-token attractor: up to 97% of its newly introduced errors land on one letter, with the attractor changing from D in Qwen to A in Llama. Logit interventions identify a dominant near-constant shift: subtracting the estimated shift on held-out items recovers 52 to 59% of lost accuracy and removes the dominant attractor, while option permutation confirms that the bias tracks token position rather than option content. With high inter-annotator agreement (κ=0.94), human-coded error-category distributions nonetheless match the weak reference on reasoning (JS = 0.00), showing that local plausibility and aggregate detectability differ. Structured tool calls exhibit more natural degradation in the Qwen family (JS = 0.09), while a globally calibrated dial transfers poorly across benchmarks (14.2-point error on two usable benchmarks). Precise score control, therefore, does not imply a portable or believable reduction in skill.
Figures & tables
Figure 1: Precise score control does not guarantee believable weakness. (A) On CapLadder reasoning, prompting, temperature, and weight noise do not combine wide coverage with calibration, whereas the weak-teacher LoRA dial spans 0.24 to 0.90 with a 1.9 -point held-out target error. (B) On items answered correctly by the base model, the naturally weak reference distributes its errors across answer positions, while a deeply throttled main SFT dial places 97% on one letter. The two systems in this panel are compared for conditional error shape, not matched accuracy. (C) The main SFT adapter has a dominant near-constant answer-token component: D is shifted by +1.14 and low-margin items flip first. Subtracting the estimated mean shift on held-out items recovers accuracy from 0.526 to 0.732 and reduces the D share from 0.70 to 0.13 .
Figure 2: Study design and empirical boundaries. We build CapLadder with four output geometries and IRT-calibrated item difficulty, apply four performance-reduction mechanisms, and audit each system against a naturally weak reference on target calibration, difficulty profile, and error distribution. The same weak-teacher dial produces different outcomes across output geometries: on multiple choice it reaches the score but converges to a cross-family answer-token attractor, whereas on the Qwen tool axis it closely matches the weak reference’s fine-grained error profile ( JS=0.09 ). Human-coded error categories match on reasoning ( JS=0.00 ; κ=0.94 ), but a naive letter histogram detects the aggregate shortcut; mean-shift cancellation weakens that audit, and a globally calibrated dial transfers with 14.2 -point error on two usable benchmarks.
axis
n
grader
answer geometry
mathematics
495
symbolic/numeric
free-form scalar
code
499
unit-test exec.
program
reasoning
492
selected token
four positions
tool
499
argument match
structured JSON
Table 1: CapLadder data card. The four axes differ in output geometry , which turns out to determine which reduction shortcut is available (§ 7 , § 8 ). Full per-source counts and licenses are in Appendix A .
method
target miss ↓
reachable acc.
wide?
prompt targeting
42 to 47 †
0.885 to 0.90
no
temperature
n/a
0.756 to 0.90
no
weight noise
n/a
0.88 to 0.90 , cliff
no
SFT dial (ours)
1.9
0.24 to 0.90
yes
Table 2: Range and calibration trade-off on CapLadder reasoning. Only the main SFT dial combines a low held-out calibration error (Eq. 4 ) with a wide reachable range. † Prompting cannot hit a requested level, so its entry is the mean absolute target miss, not an invertible calibration error. Per-axis dial curves are in Appendix C .
prompted to underperform
requested
expected
open 4B
Claude
Gemini
SFT dial (ours)
none (base)
0.90
0.902
0.968
0.976
0.902
80%
0.72
≈0.90
0.980
0.968
0.722
60%
0.54
≈0.90
0.980
0.916
0.559
40%
0.36
0.896
0.224
0.128
0.356
20%
0.18
n/a
0.012
0.012 ‡
0.237 †
Table 3: Requested versus achieved accuracy on reasoning. Each cell is the accuracy a system reaches when asked for the level in the first column. The three prompting baselines are a frontier or open model instructed to answer at that fraction of its ability; the last column is the main SFT dial calibrated to the same level. Prompting does not deliver the number: frontier models stay at the ceiling to 60% , then fall off a cliff below the answer-key floor of 0.25 and on to 0.012 ; the open model does not move. Only the dial tracks the target, to within about two points, and it saturates at the answer-position floor. The dial’s mean target miss is averaged over the in-range grid ( 80/60/40% ); the 20% request is below the answer-position floor and is out of range, not a large miss. expected is the requested fraction of the open-model base ( 0.90 ); frontier bases are ≈0.97 , so their own targets are a few points higher. † The dial saturates at its answer-position floor ( 0.237 ); ‡ Gemini reaches 0.012 at the 30% request. GPT-5.6 (completed two of six conditions) is in Appendix K . Frontier columns are Claude Opus 5 and Gemini 3.1 Pro.
method
axis
(i) score
(ii) diff.
(iii) errors
prompt targeting
reason
✗
n/a
n/a
temperature
reason
✗
n/a
n/a
weight noise
reason
✗
n/a
n/a
SFT dial (ours)
reason
✓
✗
✗
SFT dial (ours)
tool
✓
✓
✓
Table 4: Believability at the operating point matched to the weak reference’s accuracy. ✓ means the reduction matches the weak reference on that axis, ✗ that it does not. (i) score: the requested level is reachable and calibrated (held-out target miss 1.9 on reasoning, 7 to 8 on tool). (ii) difficulty: on reasoning the throttled tercile profile is flatter than the weak reference and closer to a difficulty-blind drop (Appendix F ); on tool it tracks the weak reference. (iii) errors: on reasoning a letter-frequency statistic separates the two (maximum answer-token share 0.46 versus 0.27 ) even though the Jensen-Shannon divergence is small at the exact match, and the concentration rises to 0.97 as the target drops below the weak level (Appendix G ); on tool the distributions match ( JS=0.09 ). “n/a” marks a method whose score never dropped into a comparable range, so (ii) and (iii) are not testable.
model
attractor
max share
sat. acc.
Qwen3.5-4B
D
0.97
0.237
Llama-3.2-3B
A
0.978
0.246
Llama-3.1-8B
A
0.98
0.244
weak reference
none
0.27
n/a
Table 5: The attractor reproduces across families but onto a different letter, which no property of the answer key could cause. “sat. acc.” is the accuracy at the saturated attractor, not a theoretical chance floor.
benchmark
achieved vs. target
abs. error (pt)
BFCL (transfer)
over by +17
17.4
BigCodeBench
over by +11
11.1
mean (usable)
14.2
Table 6: A single global α , calibrated to a 60% target on CapLadder reasoning, applied unchanged to two usable public benchmarks. Benchmarks whose base run is truncation-contaminated are excluded and analyzed in Appendix I .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
axis
source
n
license
math
GSM8K ( openai/gsm8k )
246
MIT
MATH ( hendrycks_math )
249
MIT
code
CodeContests
207
Apache-2.0
APPS
292
MIT
reasoning
MMLU aux. ( cais/mmlu )
294
MIT
ARC-Challenge ( ai2_arc )
198
CC BY-SA
Appendix
Table 7: CapLadder sources, counts, and licenses. All non-synthetic items are drawn from training or auxiliary splits.
α
0.0
0.5
1.0
1.5
2.0
reasoning acc.
0.887
0.887
0.826
0.676
0.498
Appendix
Table 8: Main SFT reasoning curve (calibration split). Monotone in the swept range; unlike the OPD diagnostic (Appendix M ), it does not saturate at 0.237 by α=1 .
target
target acc
α⋆
in range?
achieved
80%
0.711
1.387
yes
0.722
60%
0.533
1.901
yes
0.559
40%
0.356
>2
borderline
≈0.356
20%
0.178
–
no (below floor)
out of range
Appendix
Table 9: Reasoning target calibration. TCE =1.9 points is the mean absolute miss over the in-range grid {80%,60%} on the held-out split.
hypothesis
controlled evidence
limited conclusion
LoRA rank too small
attractor at r=8,32,64,128
rank is not the main explanation
specific family
Qwen → D ; two Llamas → A
shortcut is cross-family; token identity is model-specific
teacher error bias
teacher near uniform, student still collapses
not inherited from teacher letter frequency
output-adjacent only
LOW/HIGH matched-accuracy both >0.80
restricting layers does not remove it
multi-axis training
2-axis can be natural, 4-axis can collapse; single-axis with a distant teacher can also collapse
axis count affects the solution but is neither necessary nor sufficient
extrapolation itself
single-axis extrapolation can avoid it; the trained point can collapse
not explained by extrapolation alone
Appendix
Table 10: Controlled comparisons for the multiple-choice attractor. The teacher’s own answers are near uniform ( A / B / C / D =0.316/0.228/0.240/0.217 ; relative to gold, +1/+2/+3=0.327/0.312/0.361 ).
matched accuracy
LOW (0 to 21)
HIGH (22 to 31)
0.60
0.910
0.813
0.50
0.947
0.825
0.40
0.976
0.837
Appendix
Table 11: Max letter share at matched accuracy for the lower- and upper-layer adapters. Both collapse; restricting layers does not help.
adapter
A
B
C
D
main SFT
−0.217
−0.388
−0.182
+1.135
OPD
+0.153
−0.033
+0.038
+1.104
OPD (HIGH)
+0.002
−0.419
−0.200
+1.020
Appendix
Table 12: Mean logit shift per answer letter ( α=0.5 ) for the main SFT adapter and two OPD diagnostic adapters. The dominant systematic component is a single-token bias in every case.
system
base
thrott.
−Δ
recov.
OPD, α=0.5
0.894
0.707
0.817
59%
main SFT, α=2.0
0.894
0.526
0.732
52%
Appendix
Table 13: Counteracting the shift (estimated on one half of the items, evaluated on the other), for the main SFT adapter and the OPD control. Removing a single constant vector recovers a majority of the lost accuracy and reduces the D share from 0.70 to 0.13 ; the residual distribution remains mildly non-uniform (maximum share 0.35 ).
system
acc
easy
med
hard
corr (c,b)
base α0
0.887
0.982
0.949
0.621
−0.52
SFT α1.25 (match)
0.753
0.782
0.797
0.414
−0.35
SFT α1.5
0.676
0.745
0.729
0.345
−0.36
weak ref. (1.7B)
0.734
0.833
0.867
0.152
−0.60
Appendix
Table 14: Reasoning accuracy by difficulty tercile. At matched accuracy the throttled model ( α=1.25 ) is flatter than the weak reference: it keeps more hard items ( 0.41 vs 0.15 ) and loses more easy ones, so its difficulty profile is intermediate between the weak reference and a difficulty-blind drop.
system
acc
A / B / C / D
max
JS wk
weak ref. (1.7B)
0.734
.23/.23/.28/.26
0.28
0.00
SFT α1.25 (match)
0.753
.12/.19/.23/.46
0.46
0.036
SFT α2.0
0.498
.02/.09/.15/.74
0.74
0.199
SFT α3.5 (deep)
0.227
.00/.00/.06/.94
0.94
0.433
Appendix
Table 15: Answer-token distribution ( eabs ) versus the weak reference on reasoning, at and below matched accuracy. The gap is small at the weak reference’s own accuracy and grows as the dial is pushed lower; the deep-throttle concentrations exceed any natural respondent’s ( <0.30 ).
model
none
80%
60%
40%
30%
20%
Claude
0.968
0.980
0.980
0.224
0.052
0.012
Gemini
0.976
0.968
0.916
0.128
0.012
n/a
GPT-5.6
0.968
0.972
n/a
n/a
n/a
n/a
Appendix
Table 16: Prompted underperformance on CapLadder reasoning ( 250 items), for Claude Opus 5, Gemini 3.1 Pro, and GPT-5.6. The empirical answer-key floor is mincpc=0.237 ; scores below it (Claude at 20% , Gemini at 30% ) are certified answer avoidance.
framing
GPT-5.6
open (Qwen3.5-4B)
none
0.968
0.902
target 40%
0.976
0.896
imitate weak
0.976
n/a
answer rushed
0.976
n/a
beginner persona
0.320
0.835
Appendix
Table 17: Prompted underperformance is framing-dependent and uncalibrated. Only the beginner persona lowers the score, with errors spread across letters ( A / B / C / D =65/73/50/62 of 250 for GPT-5.6), and never to a requested target.
config
interface
Δ reason
Δ tool
#1
token, hard gate
+1.0
≈0
#2
scale, soft gate
−1.3
+1.0
#3
scale, β=0
+0.4
+1.7
Appendix
Table 18: Three GRPO configurations, reporting the change in accuracy (points) from base. All fail to move the score by more than one to two points on either axis, and #2 moves it the wrong way; β=0 removes the KL anchor and still does not help.
OPD setting
A
B
C
D
α=0.5
0.00
0.02
0.02
0.95
graded, α=1.0
0.00
0.00
0.02
0.98
Appendix
Table 19: The OPD adapter collapses onto D exactly as the main SFT dial does, despite a near-uniform teacher, and saturates at accuracy 0.237 for α≥1 .