Brevity Constraints Reverse Performance Hierarchies in Language Models

MD Azizul Hakim

cs.CLMar 11, 2026

Brevity Constraints Reverse Performance Hierarchies in Language Models

Authors: MD Azizul Hakim

Organizations: Department of Computer Science Bangladesh Sweden Polytechnic Institute Chattogram, Bangladesh

Abstract

Standard evaluation protocols reveal a counterintuitive phenomenon: on 7.7% of benchmark problems spanning five datasets, larger language models underperform smaller ones by 28.4 percentage points despite 10-100x more parameters. Through systematic evaluation of 31 models (0.5B-405B parameters) across 1,485 problems, we identify the mechanism as spontaneous scale-dependent verbosity that introduces errors through overelaboration. Causal intervention experiments demonstrate this reflects correctable prompt design rather than fundamental capability limitations. Constraining large models to produce brief responses improves accuracy by 26 percentage points and reduces performance gaps by up to two-thirds. Most critically, brevity constraints completely reverse performance hierarchies on mathematical reasoning and scientific knowledge benchmarks, with large models achieving 7.7-15.9 percentage point advantages over small models -- direct inversions of the original gaps. These reversals prove large models possess superior latent capabilities that universal prompting masks. We validate findings through three independent contamination tests and demonstrate inverse scaling operates continuously across the full parameter spectrum, with dataset-specific optimal scales ranging from 0.5B to 3.0B parameters. Our results establish that maximizing large model performance requires scale-aware prompt engineering rather than universal evaluation protocols, with immediate implications for deployment: prompt adaptation simultaneously improves accuracy and reduces computational costs.

Figures & tables

Figure 1: Problem-level performance matrix reveals discriminative inefficiency. (A) Problem categorization across five benchmarks. Orange segments highlight inverse scaling problems where small models (

\leq

10B parameters) outperform large models (

\geq

70B parameters). (B) Overall distribution shows 7.7% inverse scaling rate across 1,485 problems with Cohen’s

d=1.34

effect size.

Figure 2: Discovery of systematic inverse scaling across benchmarks. (A) Prevalence ranges from 3.9% (MMLU-STEM) to 11.3% (BoolQ), with 115 total inverse problems. (B) Performance gap distribution shows mean 28.4pp advantage for small models (

\leq

10B). (C) Strong negative correlation between model size and accuracy on inverse problems; small models achieve 66.1% vs large models’ 41.5%.

Figure 3: Causal evidence: brevity constraints eliminate inverse scaling. (A) Performance across three conditions shows large models improve dramatically under brevity constraints (Control: 40.2% → Brief: 66.5%, +26.3pp), reducing gap by 67% (44.2pp → 14.8pp,

t=7.80

p<0.0001

). (B) Gap reduction varies by dataset, with complete reversals in GSM8K and MMLU-STEM where brief condition causes large models to outperform. (C) Response length validation confirms intervention successfully manipulated verbosity (Control: 197 tokens → Brief: 78 tokens, 60% reduction), establishing causal link between overthinking and performance degradation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Contamination analysis through three independent validation tests. (A) Response diversity heatmap shows 89–100% unique responses across datasets, contradicting memorization patterns which would produce template-like responses. (B) Length variability measured by coefficient of variation (CV) ranges from 0.31 to 1.21, with all datasets exceeding memorization threshold (CV $<$ 0.15) and 3/5 exceeding natural variation threshold (CV $>$ 0.30). (C) Error pattern classification reveals over-reasoning (verbose incorrect logic) as dominant failure mode (41–82% of large model failures), inconsistent with memorization hypothesis which predicts either correct retrieval or evasive brevity. Convergent evidence across all tests supports genuine capability differences rather than contamination artifacts.

Figure 5: Dataset-specific breakdown reveals heterogeneous inverse scaling patterns. For each benchmark: (Left) Problem-level accuracy heatmap comparing small models (top) versus large models (bottom), with inverse scaling problems marked by orange dashed lines. (Middle) Model family performance ranked by accuracy, colored by size (blue: $\leq$ 10B, red: $\geq$ 70B). (Right) Response length distributions for normal versus inverse problems. Results show: (1) inverse scaling occurs across all task types, (2) small models consistently outperform large models on inverse problems

Figure 6: Inverse scaling is architecture-independent across model families. (A) Performance comparison across four major families (Llama, Qwen, Gemma, Mistral) on all five datasets shows consistent inverse scaling patterns independent of architecture. Gemma achieves highest accuracy (52–94%), followed by Qwen (40–83%), Mistral (18–89%), and Llama (41–68%), yet all families exhibit the phenomenon. (B) Model size ranges demonstrate each family spans small to large parameters: Llama (2B–405B), Qwen (0.5B–32B), Gemma (1B–9B), Mistral (7B–24B), ensuring within-family scale comparisons. (C) Perfect cross-dataset consistency (5/5) for all families confirms inverse scaling transcends architectural variations including attention mechanisms (Llama standard, Mistral grouped-query), training objectives, and design optimizations (Gemma efficiency-focused). Convergent evidence rules out architecture-specific artifacts and establishes inverse scaling as fundamental property of scale rather than design.

Test	Dataset	Statistic	p-value	Effect Size	CI (95%)
Mann-Whitney U Tests (Small vs Large on Inverse Problems)
MW-U	Overall	–	$<$ 0.001	$d=1.34$	[0.89, 1.79]
MW-U	GSM8K	4125.0	$<$ 0.001	–	[0.21, 0.31]
MW-U	BoolQ	3021.0	$<$ 0.001	–	[0.25, 0.35]
MW-U	ARC-Easy	5135.0	$<$ 0.001	–	[0.23, 0.33]
MW-U	CommonsenseQA	3820.0	$<$ 0.001	–	[0.24, 0.34]

Appendix

Table 1: Complete statistical test results across all analyses. All tests demonstrate highly significant effects supporting inverse scaling phenomenon and causal intervention efficacy.

Model	Params	Family	Release Date	Context
Llama Family (Meta)
Llama-2-13B	13.0B	Llama-2	Jul 2023	4K
Llama-3-70B-Base	70.0B	Llama-3	Apr 2024	8K
Llama-3-70B-Instruct	70.0B	Llama-3	Apr 2024	8K
Llama-3.1-8B-Instruct	8.0B	Llama-3.1	Jul 2024	128K
Llama-3.1-405B-Instruct	405.0B	Llama-3.1	Jul 2024	128K

Appendix

Table 2: Complete specifications for 31 analyzed models. All models evaluated with greedy decoding ( do_sample=False , nucleus sampling disabled). Parameter counts and release dates from official model cards and technical reports.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Feb 17, 2026Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1Language Model Performance Prediction Language Model Scaling Laws
Harvard University · Stanford University
Where Larger Models Excel: The Primacy of Constraint-Guided Reasoning
May 9, 2026Guan-Yi Lin, Hen-Hsen HuangLanguage Model Scaling Laws LLM Reasoning
National Chengchi University Taipei, Taiwan · Academia Sinica Taipei, Taiwan
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University

Feb 17, 2026J/K move · Enter open · S save

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade

Harvard University · Stanford University

Language Model Performance Prediction Language Model Scaling Laws LLM Evaluation

Open paper AskPDF arXiv

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

Abstract

LLM Evaluation

Explore similar work

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Where Larger Models Excel: The Primacy of Constraint-Guided Reasoning

Brittlebench: Quantifying LLM robustness via prompt sensitivity