System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.
Figures & tables
Figure 1. How System One models rerank and what we compare. (a) For each BM25 candidate, a decision model is asked whether the passage answers the query. The candidates are sorted by the probability of “yes”. The input of the model (state) holds one passage (isolated mode) or all 100 candidates (joint mode). (b) On one Qwen backbone, the untuned model, its decision fine-tunes and its reranker fine-tunes rerank the same candidates and are compared on ranking and on calibration. Two panels. Panel a: a query with one passage per state (isolated mode) or with all 100 candidate passages in one state (joint mode) goes into a decision model together with the question ``Does document doc answer the query?'', asked once per passage. The model returns a probability of yes for every passage, shown as bars on example passages in BM25 order, and sorting by this probability gives the reranked list. Panel b: one Qwen backbone of 0.6B to 27B parameters branches into three variants: the untuned model asked zero-shot, a decision fine-tune (Kev, Clef) and a reranker fine-tune (Qwen3-Reranker, Prism, ours). All three read the same BM25 top-100 of TREC DL 2019–2023 and five BEIR collections and return a probability of yes, which is evaluated for ranking (nDCG@10) and for calibration (predicted probability against observed relevance).