Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation
Organizations: Jožef Stefan Institute, Ljubljana, Slovenia
Abstract
Multi-agent systems of language models are often motivated by an averaging argument: if the agents' errors are unbiased and independent, the error of an aggregate of N answers falls as 1/sqrt(N). Fermi estimation, whose continuous answers are aggregated on the logarithmic scale, permits a direct test. We decompose the log10 error of each answer into the mean error of its model on the item, common to all its draws on the item, and a residual, and estimate both from 2.51M answers given before any communication by 15 open-weight models (3B-72B) on realFP, with replication under a single prompt pair on realFP, Science Olympiad Fermi questions and a guesstimation set. Given the item, teammates' errors are uncorrelated and teams behave as independent draws, but the item-level component accounts for a median 0.906 of the error variance over items. Teams of thirty therefore reduce a single agent's error by a median 1.7% (at most 5.4%), whereas draws that were also unbiased on each item would reduce it by 74-79%. All models compress their estimates toward about 10^6; this trend explains a median 17% of the item-level component, and a cross-validated linear recalibration on labelled items removes the overall bias but not that component. Three rounds of debate reduce within-team dispersion by 93% and leave the item-level component unchanged; revision without peers attains the same small gain. Mixing five models of similar accuracy reduces error by 6.6% relative to their own homogeneous teams, but no mixture improves on the most accurate model. Higher sampling temperature changes team error by about 1% at most, and contrastive activation steering not detectably; per-agent scaffold prompts (not significant after multiplicity correction) and skipping one layer in a single-model screen improve it. The results are conditional on the models, benchmarks, prompts and settings tested.
Figures & tables
| corpus | models | items | rounds | prompts | used in | |
| main sweep | 15 | realFP 529 | 2–30 | 1–3, genuine peers | both pairs | § 4 – 6 |
| second corpus | 15 | realFP, SciOly 500, Guess. 15 | 2–30 | 1–3, peers and self-revision | revised | § 4 – 6 |
| heterogeneous pool | 5 | realFP 529 | 3, 5 | 1–3, three peer sources | revised + arms | § 6 , 7 |
| temperature sweep | 4 | realFP 529 | 10 | 1 | revised | § 7 |
| activation screen | 1 | realFP 150 | 5 | 1 | revised | § 7 |
| realFP (main) | realFP (second) | SciOly (second) | Guesstimate (second) | |
| models | 15 | 15 | 15 | 14 |
| items | 529 | 529 | 500 | 15 |
| one agent within an order of magnitude (%) | 12–34 | 14–34 | 15–46 | 3–67 |
| one agent off by more than (%) | 30–63 | 31–57 | 17–54 | 0–80 |
| models with mean error | 10/15 | 9/15 | 5/15 | 12/14 |
| bias shrinkage across models (realised / ) | 1.00 / 3.87 | 1.04 / 3.87 | 1.65 / 2.83 | 1.00 / 3.74 |
| heterogeneous pool (529 items) | homogeneous teams (529 items) | |||
| contrast | 95% CI | 95% CI | ||
| genuine peers, round 3 round 1 | ||||
| self-revision, round 3 round 1 | ||||
| peers from other items, round 3 round 1 | – | |||
| round 3: genuine peers other items | – | |||
| round 3: genuine peers self-revision | ||||
| intervention | against | items | team MAE@5 [95% CI] | (%) | ||
| Composition (realFP main corpus, 526 items; five models of similar accuracy, ) | ||||||
| mixture of the five | their own teams | 5 | 526 | – | – | |
| mixture of the five | best member’s team | 5 | 526 | – | – | |
| Prompt level (heterogeneous five-model pool, realFP) | ||||||
| per-agent anchors † | baseline | 5 | 529 | 0.747 | ||
| per-agent scaffolds | baseline | 5 | 529 | 0.089 | ||
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| short name | family | params | precision | ctx | round-1 answers |
| qwen2.5-3b | Qwen2.5 | 3.1B | bf16 | 16384 | 182,938 |
| smollm3-3b | SmolLM3 | 3.1B | bf16 | 8192 | 185,677 |
| qwen3-4b | Qwen3 | 4.0B | bf16 | 16384 | 184,977 |
| mistral-7b | Mistral | 7.2B | bf16 | 16384 | 183,330 |
| olmo2-7b | OLMo 2 | 7.3B | bf16 | 8192 | 185,679 |
| qwen2.5-7b | Qwen2.5 | 7.6B | bf16 | 16384 | 185,683 |
| runs | round-1 MAE@5 | bias | IQR, | debate, round 3 round 1 | |||||||
| model | S/D | S | D | D S | S | D | S | D | S | D | D S |
| llama3.1-8b | 2/1 | 2.99 | 2.63 | 0.84 | 0.75 | ||||||
| mistral-7b | 2/1 | 2.95 | 2.74 | 0.03 | 0.14 | ||||||
| phi4-14b | 2/1 | 2.20 | 2.13 | 0.30 | 0.01 | ||||||
| qwen3-14b | 2/1 | 2.10 | 2.04 | 0.00 | 0.00 | ||||||
| qwen3-8b | 2/1 | 2.16 | 2.08 | 0.00 | 0.00 | ||||||
| coverage | error (OOM) | accuracy bands (%) | ||||||||
| model | items | answers | parse | med. | med. | MAE@5 | ||||
| qwen2.5-3b | 529 | 182,938 | 98.5 | 2.40 | 2.60 0.07 | 13.8 | 25.6 | 55.5 | 10.7 | |
| smollm3-3b | 529 | 185,677 | 100.0 | 2.26 | 2.53 0.06 | 15.4 | 27.6 | 54.0 | 10.7 | |
| qwen3-4b | 527 | 184,977 | 99.6 | 2.12 | 2.47 0.08 | 16.8 | 28.2 | 51.8 | 9.6 | |
| mistral-7b | 529 | 183,330 | 98.7 | 2.72 | 2.87 0.07 | 13.6 | 23.2 | 59.8 | 20.1 | |
| olmo2-7b | 529 | 185,679 | 100.0 | 1.79 | 2.27 0.07 | 19.3 | 34.2 | 45.7 | 9.9 | |
| model | items | slope | crossing | |
| qwen2.5-3b | 486 | |||
| smollm3-3b | 486 | |||
| qwen3-4b | 484 | |||
| mistral-7b | 486 | |||
| olmo2-7b | 486 | |||
| qwen2.5-7b | 486 |
| given the item | pooled | |||||||
| model | one agent | team | gain [95% CI] | resampled | unbiased | unbiased | % realised | |
| gpt-oss-20b | 30 | 1.693 | 1.602 | 1.602 | 0.407 | 0.837 | ||
| marin-8b | 30 | 2.499 | 2.374 | 2.376 | 0.570 | 0.978 | ||
| llama3.1-8b | 30 | 2.881 | 2.793 | 2.740 | 0.652 | 1.065 | ||
| smollm3-3b | 30 | 2.527 | 2.450 | 2.411 | 0.571 | 0.983 | ||
| qwen2.5-7b | 30 | 2.323 | 2.266 | 2.263 | 0.504 | 0.973 | ||
| model | (OOM) | run (%) | |||||
| qwen2.5-3b | 0.882 | 1.10 | 1.13 | 1.13 | 2.87 | 0.881 | 0.1 |
| smollm3-3b | 0.867 | 1.12 | 1.15 | 1.15 | 2.76 | 0.789 | 8.1 |
| qwen3-4b | 0.964 | 1.03 | 1.04 | 1.04 | 2.83 | 0.962 | 0.1 |
| mistral-7b | 0.931 | 1.06 | 1.07 | 1.07 | 3.05 | 0.815 | 10.6 |
| olmo2-7b | 0.922 | 1.07 | 1.08 | 1.09 | 2.74 | 0.916 | 0.0 |
| qwen2.5-7b | 0.911 | 1.08 | 1.09 | 1.10 | 2.80 | 0.914 | 0.1 |
| realFP (main) | realFP (second) | SciOly (second) | Guesstimate (second) | |||||
| G | M | G | M | G | M | G | M | |
| error reduction at , real teams (% of one agent; median / max) | 1.7 / 5.4 | 1.0 / 5.6 | 1.7 / 6.0 | 1.1 / 5.8 | 1.0 / 5.7 | 1.4 / 10.1 | 2.1 / 21.7 | 1.1 / 35.5 |
| mixtures significantly better than the best homogeneous team | 0/4,928 | 1/4,928 | 0/4,928 | 0/4,928 | 0/4,928 | 0/4,928 | 0/3,458 | 0/3,458 |
| debate: total gain over one agent (%) | 2.1 | 1.7 | 2.4 | 2.1 | 5.5 | 5.3 | 8.0 | 6.4 |
| debate: models helped / harmed (significant) | 8/4 | 9/3 | 8/2 | 9/3 | 10/0 | 9/0 | 5/3 | 3/2 |
| round 3: genuine peers self-revision (OOM) | – | – | ||||||
| (OOM) | (OOM 2 ) | (OOM) | team MAE@5 | |||||
| model | r1 | r3 | r1 | r3 | r1 | r3 | r1 | r3 |
| qwen2.5-3b | 2.87 | 2.86 | 1.08 | 0.48 | 0.43 | 0.00 | 2.59 | 2.48 |
| smollm3-3b | 2.62 | 2.64 | 1.10 | 0.88 | 0.51 | 0.06 | 2.47 | 2.49 |
| qwen3-4b | 2.84 | 2.82 | 0.31 | 0.22 | 0.00 | 0.00 | 2.46 | 2.41 |
| mistral-7b | 2.85 | 2.85 | 0.83 | 0.74 | 0.06 | 0.00 | 2.84 | 2.82 |
| olmo2-7b | 2.74 | 2.72 | 0.66 | 0.59 | 0.08 | 0.00 | 2.23 | 2.31 |
| arm | ladder | team MAE@5 | (%) | parse (%) | ||||
| grid | 5 | 5.12 | 3.98 | 0.563 | 0.791 | 2.26 | 51.5 | 99.7 |
| anchor | 5 | 5.09 | 4.06 | 0.556 | 0.770 | 2.25 | 51.7 | 99.7 |
| scaffold | 5 | 4.62 | 4.53 | 0.505 | 0.753 | 2.16 | 49.4 | 97.9 |
| logcalc | 3 | 2.55 | 10.61 | 0.194 | 0.154 | 2.94 | 66.9 | 93.2 |
| anchor_logcalc | 5 | 2.47 | 10.68 | 0.188 | 0.178 | 2.89 | 66.9 | 86.7 |
| grounding_logcalc | 3 | 2.78 | 11.47 | 0.195 | 0.265 | 3.20 | 70.8 | 89.7 |
| model | one agent | team | |||||
| qwen3-4b | 0.4 | 7.09 | 0.432 | 0.943 | 0.000 | 2.162 | 2.128 |
| qwen3-4b | 0.8 | 6.84 | 0.672 | 0.910 | 0.168 | 2.164 | 2.116 |
| qwen3-4b | 1.0 | 6.83 | 0.828 | 0.892 | 0.378 | 2.188 | 2.126 |
| qwen3-8b | 0.4 | 7.08 | 0.268 | 0.964 | 0.000 | 2.085 | 2.083 |
| qwen3-8b | 0.8 | 6.92 | 0.460 | 0.938 | 0.301 | 2.090 | 2.072 |
| qwen3-8b | 1.0 | 6.82 | 0.537 | 0.927 | 0.358 | 2.086 | 2.065 |
| arm | (%) | team | 95% CI | verdict | ||
| headmask | 0.5 | lower , team n.s. | ||||
| headmask | 1.0 | no effect | ||||
| headmask | 2.0 | no effect | ||||
| layerskip | 0.5 | team better | ||||
| layerskip | 1.0 | team better | ||||
| layerskip | 2.0 | team better |
| model | items | exact OOM (%) | MAE@5 | mean bias | slope | % of ref. gain | peers self | ||
| qwen2.5-3b | 499 | 15 | 2.65 | 0.866 | 1.15 | ||||
| smollm3-3b | 500 | 18 | 2.49 | 0.890 | 1.12 | ||||
| qwen3-4b | 499 | 20 | 2.39 | 0.944 | 1.06 | ||||
| mistral-7b | 498 | 19 | 2.60 | 0.890 | 1.12 | ||||
| olmo2-7b | 500 | 17 | 2.58 | 0.906 | 1.10 | ||||
| qwen2.5-7b | 500 | 20 | 2.28 | 0.906 | 1.10 |
| model | items | exact OOM (%) | MAE@5 | mean bias | slope | % of ref. gain | peers self | ||
| qwen2.5-3b | 15 | 10 | 2.20 | 0.883 | 1.13 | ||||
| smollm3-3b | 15 | 26 | 1.52 | 0.633 | 1.55 | ||||
| qwen3-4b | 15 | 32 | 0.84 | 0.962 | 1.04 | ||||
| mistral-7b | 15 | 10 | 1.54 | 0.707 | 1.40 | ||||
| olmo2-7b | 15 | 6 | 1.93 | 0.626 | 1.57 | ||||
| qwen2.5-7b | 15 | 25 | 1.10 | 0.715 | 1.38 |
Language Model Ensembles
Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.
See all 154 papers