Organizations: Independent Researcher · University of Chinese Academy of Sciences · Pusan National University · Shenzhen University of Advanced Technology
A benchmark score is commonly read as a proxy for model capability. We study the inverse problem: can a model be driven to a target score, and do its failures resemble those of a naturally weaker model? We operationalize believable performance reduction along three axes: target calibration, item-difficulty structure, and error-distribution similarity to a real weak reference. We introduce CapLadder, an IRT-calibrated suite of 1,985 items spanning mathematics, code, multiple-choice reasoning, and tool use. A simple weak-teacher LoRA dial reaches unseen target levels on reasoning with a 1.9-point held-out calibration error, whereas prompting, temperature, and weight noise cannot combine comparable range and precision. Yet on multiple choice, the dial converges to an answer-token attractor: up to 97% of its newly introduced errors land on one letter, with the attractor changing from D in Qwen to A in Llama. Logit interventions identify a dominant near-constant shift: subtracting the estimated shift on held-out items recovers 52 to 59% of lost accuracy and removes the dominant attractor, while option permutation confirms that the bias tracks token position rather than option content. With high inter-annotator agreement (κ=0.94), human-coded error-category distributions nonetheless match the weak reference on reasoning (JS = 0.00), showing that local plausibility and aggregate detectability differ. Structured tool calls exhibit more natural degradation in the Qwen family (JS = 0.09), while a globally calibrated dial transfers poorly across benchmarks (14.2-point error on two usable benchmarks). Precise score control, therefore, does not imply a portable or believable reduction in skill.
Figures & tables
Figure 1: Precise score control does not guarantee believable weakness. (A) On CapLadder reasoning, prompting, temperature, and weight noise do not combine wide coverage with calibration, whereas the weak-teacher LoRA dial spans 0.24 to 0.90 with a 1.9 -point held-out target error. (B) On items answered correctly by the base model, the naturally weak reference distributes its errors across answer positions, while a deeply throttled main SFT dial places 97% on one letter. The two systems in this panel are compared for conditional error shape, not matched accuracy. (C) The main SFT adapter has a dominant near-constant answer-token component: D is shifted by +1.14 and low-margin items flip first. Subtracting the estimated mean shift on held-out items recovers accuracy from 0.526 to 0.732 and reduces the D share from 0.70 to 0.13 .
Figure 2: Study design and empirical boundaries. We build CapLadder with four output geometries and IRT-calibrated item difficulty, apply four performance-reduction mechanisms, and audit each system against a naturally weak reference on target calibration, difficulty profile, and error distribution. The same weak-teacher dial produces different outcomes across output geometries: on multiple choice it reaches the score but converges to a cross-family answer-token attractor, whereas on the Qwen tool axis it closely matches the weak reference’s fine-grained error profile ( JS=0.09 ). Human-coded error categories match on reasoning ( JS=0.00 ; κ=0.94 ), but a naive letter histogram detects the aggregate shortcut; mean-shift cancellation weakens that audit, and a globally calibrated dial transfers with 14.2 -point error on two usable benchmarks.
axis
n
grader
answer geometry
mathematics
495
symbolic/numeric
free-form scalar
code
499
unit-test exec.
program
reasoning
492
selected token
four positions
tool
499
argument match
structured JSON
Table 1: CapLadder data card. The four axes differ in output geometry , which turns out to determine which reduction shortcut is available (§ 7 , § 8 ). Full per-source counts and licenses are in Appendix A .
method
target miss ↓
reachable acc.
wide?
prompt targeting
42 to 47 †
0.885 to 0.90
no
temperature
n/a
0.756 to 0.90
no
weight noise
n/a
0.88 to 0.90 , cliff
no
SFT dial (ours)
1.9
0.24 to 0.90
yes
Table 2: Range and calibration trade-off on CapLadder reasoning. Only the main SFT dial combines a low held-out calibration error (Eq. 4 ) with a wide reachable range. † Prompting cannot hit a requested level, so its entry is the mean absolute target miss, not an invertible calibration error. Per-axis dial curves are in Appendix C .
prompted to underperform
requested
expected
open 4B
Claude
Gemini
SFT dial (ours)
none (base)
0.90
0.902
0.968
0.976
0.902
80%
0.72
≈0.90
0.980
0.968
0.722
60%
0.54
≈0.90
0.980
0.916
0.559
40%
0.36
0.896
0.224
0.128
0.356
20%
0.18
n/a
0.012
0.012 ‡
0.237 †
Table 3: Requested versus achieved accuracy on reasoning. Each cell is the accuracy a system reaches when asked for the level in the first column. The three prompting baselines are a frontier or open model instructed to answer at that fraction of its ability; the last column is the main SFT dial calibrated to the same level. Prompting does not deliver the number: frontier models stay at the ceiling to 60% , then fall off a cliff below the answer-key floor of 0.25 and on to 0.012 ; the open model does not move. Only the dial tracks the target, to within about two points, and it saturates at the answer-position floor. The dial’s mean target miss is averaged over the in-range grid ( 80/60/40% ); the 20% request is below the answer-position floor and is out of range, not a large miss. expected is the requested fraction of the open-model base ( 0.90 ); frontier bases are ≈0.97 , so their own targets are a few points higher. † The dial saturates at its answer-position floor ( 0.237 ); ‡ Gemini reaches 0.012 at the 30% request. GPT-5.6 (completed two of six conditions) is in Appendix K . Frontier columns are Claude Opus 5 and Gemini 3.1 Pro.
method
axis
(i) score
(ii) diff.
(iii) errors
prompt targeting
reason
✗
n/a
n/a
temperature
reason
✗
n/a
n/a
weight noise
reason
✗
n/a
n/a
SFT dial (ours)
reason
✓
✗
✗
SFT dial (ours)
tool
✓
✓
✓
Table 4: Believability at the operating point matched to the weak reference’s accuracy. ✓ means the reduction matches the weak reference on that axis, ✗ that it does not. (i) score: the requested level is reachable and calibrated (held-out target miss 1.9 on reasoning, 7 to 8 on tool). (ii) difficulty: on reasoning the throttled tercile profile is flatter than the weak reference and closer to a difficulty-blind drop (Appendix F ); on tool it tracks the weak reference. (iii) errors: on reasoning a letter-frequency statistic separates the two (maximum answer-token share 0.46 versus 0.27 ) even though the Jensen-Shannon divergence is small at the exact match, and the concentration rises to 0.97 as the target drops below the weak level (Appendix G ); on tool the distributions match ( JS=0.09 ). “n/a” marks a method whose score never dropped into a comparable range, so (ii) and (iii) are not testable.
model
attractor
max share
sat. acc.
Qwen3.5-4B
D
0.97
0.237
Llama-3.2-3B
A
0.978
0.246
Llama-3.1-8B
A
0.98
0.244
weak reference
none
0.27
n/a
Table 5: The attractor reproduces across families but onto a different letter, which no property of the answer key could cause. “sat. acc.” is the accuracy at the saturated attractor, not a theoretical chance floor.
benchmark
achieved vs. target
abs. error (pt)
BFCL (transfer)
over by +17
17.4
BigCodeBench
over by +11
11.1
mean (usable)
14.2
Table 6: A single global α , calibrated to a 60% target on CapLadder reasoning, applied unchanged to two usable public benchmarks. Benchmarks whose base run is truncation-contaminated are excluded and analyzed in Appendix I .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
axis
source
n
license
math
GSM8K ( openai/gsm8k )
246
MIT
MATH ( hendrycks_math )
249
MIT
code
CodeContests
207
Apache-2.0
APPS
292
MIT
reasoning
MMLU aux. ( cais/mmlu )
294
MIT
ARC-Challenge ( ai2_arc )
198
CC BY-SA
Appendix
Table 7: CapLadder sources, counts, and licenses. All non-synthetic items are drawn from training or auxiliary splits.
α
0.0
0.5
1.0
1.5
2.0
reasoning acc.
0.887
0.887
0.826
0.676
0.498
Appendix
Table 8: Main SFT reasoning curve (calibration split). Monotone in the swept range; unlike the OPD diagnostic (Appendix M ), it does not saturate at 0.237 by α=1 .
target
target acc
α⋆
in range?
achieved
80%
0.711
1.387
yes
0.722
60%
0.533
1.901
yes
0.559
40%
0.356
>2
borderline
≈0.356
20%
0.178
–
no (below floor)
out of range
Appendix
Table 9: Reasoning target calibration. TCE =1.9 points is the mean absolute miss over the in-range grid {80%,60%} on the held-out split.
hypothesis
controlled evidence
limited conclusion
LoRA rank too small
attractor at r=8,32,64,128
rank is not the main explanation
specific family
Qwen → D ; two Llamas → A
shortcut is cross-family; token identity is model-specific
teacher error bias
teacher near uniform, student still collapses
not inherited from teacher letter frequency
output-adjacent only
LOW/HIGH matched-accuracy both >0.80
restricting layers does not remove it
multi-axis training
2-axis can be natural, 4-axis can collapse; single-axis with a distant teacher can also collapse
axis count affects the solution but is neither necessary nor sufficient
extrapolation itself
single-axis extrapolation can avoid it; the trained point can collapse
not explained by extrapolation alone
Appendix
Table 10: Controlled comparisons for the multiple-choice attractor. The teacher’s own answers are near uniform ( A / B / C / D =0.316/0.228/0.240/0.217 ; relative to gold, +1/+2/+3=0.327/0.312/0.361 ).
matched accuracy
LOW (0 to 21)
HIGH (22 to 31)
0.60
0.910
0.813
0.50
0.947
0.825
0.40
0.976
0.837
Appendix
Table 11: Max letter share at matched accuracy for the lower- and upper-layer adapters. Both collapse; restricting layers does not help.
adapter
A
B
C
D
main SFT
−0.217
−0.388
−0.182
+1.135
OPD
+0.153
−0.033
+0.038
+1.104
OPD (HIGH)
+0.002
−0.419
−0.200
+1.020
Appendix
Table 12: Mean logit shift per answer letter ( α=0.5 ) for the main SFT adapter and two OPD diagnostic adapters. The dominant systematic component is a single-token bias in every case.
system
base
thrott.
−Δ
recov.
OPD, α=0.5
0.894
0.707
0.817
59%
main SFT, α=2.0
0.894
0.526
0.732
52%
Appendix
Table 13: Counteracting the shift (estimated on one half of the items, evaluated on the other), for the main SFT adapter and the OPD control. Removing a single constant vector recovers a majority of the lost accuracy and reduces the D share from 0.70 to 0.13 ; the residual distribution remains mildly non-uniform (maximum share 0.35 ).
system
acc
easy
med
hard
corr (c,b)
base α0
0.887
0.982
0.949
0.621
−0.52
SFT α1.25 (match)
0.753
0.782
0.797
0.414
−0.35
SFT α1.5
0.676
0.745
0.729
0.345
−0.36
weak ref. (1.7B)
0.734
0.833
0.867
0.152
−0.60
Appendix
Table 14: Reasoning accuracy by difficulty tercile. At matched accuracy the throttled model ( α=1.25 ) is flatter than the weak reference: it keeps more hard items ( 0.41 vs 0.15 ) and loses more easy ones, so its difficulty profile is intermediate between the weak reference and a difficulty-blind drop.
system
acc
A / B / C / D
max
JS wk
weak ref. (1.7B)
0.734
.23/.23/.28/.26
0.28
0.00
SFT α1.25 (match)
0.753
.12/.19/.23/.46
0.46
0.036
SFT α2.0
0.498
.02/.09/.15/.74
0.74
0.199
SFT α3.5 (deep)
0.227
.00/.00/.06/.94
0.94
0.433
Appendix
Table 15: Answer-token distribution ( eabs ) versus the weak reference on reasoning, at and below matched accuracy. The gap is small at the weak reference’s own accuracy and grows as the dial is pushed lower; the deep-throttle concentrations exceed any natural respondent’s ( <0.30 ).
model
none
80%
60%
40%
30%
20%
Claude
0.968
0.980
0.980
0.224
0.052
0.012
Gemini
0.976
0.968
0.916
0.128
0.012
n/a
GPT-5.6
0.968
0.972
n/a
n/a
n/a
n/a
Appendix
Table 16: Prompted underperformance on CapLadder reasoning ( 250 items), for Claude Opus 5, Gemini 3.1 Pro, and GPT-5.6. The empirical answer-key floor is mincpc=0.237 ; scores below it (Claude at 20% , Gemini at 30% ) are certified answer avoidance.
framing
GPT-5.6
open (Qwen3.5-4B)
none
0.968
0.902
target 40%
0.976
0.896
imitate weak
0.976
n/a
answer rushed
0.976
n/a
beginner persona
0.320
0.835
Appendix
Table 17: Prompted underperformance is framing-dependent and uncalibrated. Only the beginner persona lowers the score, with errors spread across letters ( A / B / C / D =65/73/50/62 of 250 for GPT-5.6), and never to a requested target.
config
interface
Δ reason
Δ tool
#1
token, hard gate
+1.0
≈0
#2
scale, soft gate
−1.3
+1.0
#3
scale, β=0
+0.4
+1.7
Appendix
Table 18: Three GRPO configurations, reporting the change in accuracy (points) from base. All fail to move the score by more than one to two points on either axis, and #2 moves it the wrong way; β=0 removes the KL anchor and still does not help.
OPD setting
A
B
C
D
α=0.5
0.00
0.02
0.02
0.95
graded, α=1.0
0.00
0.00
0.02
0.98
Appendix
Table 19: The OPD adapter collapses onto D exactly as the main SFT dial does, despite a near-uniform teacher, and saturates at accuracy 0.237 for α≥1 .
Existing benchmarks typically report accuracy for a single model on a single run. This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions: (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. To quantify this gap, we introduce the Capability Frontier: a Pareto frontier over a set of models that characterizes the best achievable performance at each cost level under optimal selection across models and generations (i.e., via an oracle). Our construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples. We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, comparing Capability Frontier performance at matched cost to each benchmark's top-performing model. Correcting for single-model evaluation yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with SOTA accuracy matched at 85% cost reduction. Complementing these empirical results, we use controlled probabilistic simulations to show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model. Our findings suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings.
Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.
Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability θ preserves the global performance structure. At the same time, θ provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.