Efficient Best-Of-N with Radial Consensus Score

Manh Nguyen; Sunil Gupta; Hung Le

cs.CLApr 14, 2026

Efficient Best-Of-N with Radial Consensus Score

Authors: Manh Nguyen, Sunil Gupta, Hung Le

Organizations: Applied Artificial Intelligence Initiative, Deakin University, Australia

Abstract

Solving complex problems with large language models (LLMs) often requires selecting the best answer from multiple candidate responses (best-of- $N$ ), yet identifying the most reliable one remains challenging, particularly when correctness does not align with majority agreement. Existing approaches based on voting or probability scoring rely on surface-level signals (e.g., frequency or token-level likelihood), often missing high-quality rare answers and ignoring the geometric structure of candidate representations, where correctness may emerge as semantic consistency rather than surface agreement. To address these limitations, we introduce Radial Consensus Score (RCS), a simple, efficient, and training-free method for best-of- $N$ selection. RCS models semantic consensus by computing a weighted Frechet mean (semantic center) of answer embeddings and ranking candidates by their radial distance to this center. This enables it to recover semantically coherent correct answers despite lexical fragmentation, even when no single surface form holds a plurality under exact-match voting. Importantly, RCS offers a flexible framework that supports multiple weighting schemes, enabling the integration of arbitrary signals, such as semantic agreement or model confidence, with the position of the center adapting accordingly. Extensive experiments across five open-weight models and various benchmarks demonstrate that RCS matches or outperforms strong baselines at every sampling budget, with the largest gains when correct answers are lexically fragmented. Moreover, RCS serves as an effective drop-in replacement for majority voting in multi-agent debate and remains effective in black-box scenarios. These results suggest that geometric consensus can serve as a scalable and reliable alternative to voting-based methods for answer selection. Code is publicly available at https://github.com/manhitv/RCS.

Figures & tables

Figure 1: (a) Majority voting ignores semantically similar answers by treating surface forms independently. (b) It also fails to select high-confidence but minority answers. RCS addresses the lexical fragmentation issue by mapping answers into an embedding space via Encoder

\mathbf{E}

and estimating a centroid

\mathbf{c}

, either uniformly or probability-weighted over answer embeddings; the latter can also recover high-confidence minority answers. Additional qualitative examples are provided in Section 5.3 .

Variant	Distribution $P$	Supported LLMs
Uniform	$p_{i}=\frac{1}{N}$	Black-box
Frequency-weighted	$p_{i}=\frac{f_{i}}{\sum_{j}f_{j}}$	Black-box
Cosine-weighted	$p_{i}\propto\frac{1}{N}\sum_{j=1}^{N}\cos(\mathbf{u}_{i},\mathbf{u}_{j})$	Black-box
Probability-weighted	$p_{i}\propto\Pr(y_{i}\|\mathbf{x})$	White-box

Table 1: Different instantiations of the distribution

P

in RCS, where

f_{i}

and

\Pr(y_{i}|\mathbf{x})

denote the empirical frequency and model probability of candidate

y_{i}

, respectively. For the cosine-similarity-weighted variant,

\cos(\mathbf{u}_{i},\mathbf{u}_{j})

denotes the cosine similarity between

\ell_{2}

-normalized embeddings.

Model	Method	BBF	SciQ	GPQA	Arithmetics	GSM8K	Form.Log.	Average
Qwen2.5-3B	Greedy	✓	59.3	20.9	89.0	65.0	34.9	53.8
	Oracle	✓	80.8 $\pm$ 0.7	51.0 $\pm$ 2.9	99.5 $\pm$ 0.5	93.1 $\pm$ 0.8	73.5 $\pm$ 5.7	79.6
	NLL	✗	59.4 $\pm$ 0.7	15.7 $\pm$ 2.4	36.0 $\pm$ 0.9	25.0 $\pm$ 3.1	23.0 $\pm$ 2.7	31.8
	ANLL	✗	59.2 $\pm$ 0.5	15.9 $\pm$ 2.2	36.0 $\pm$ 1.3	24.9 $\pm$ 3.3	22.8 $\pm$ 3.0	31.8
	SC	✓	64.4 $\pm$ 0.6	21.4 $\pm$ 0.6	78.7 $\pm$ 3.3	52.8 $\pm$ 1.3	40.5 $\pm$ 2.1	51.6
	SCW	✓	64.5 $\pm$ 0.8	23.0 $\pm$ 0.3	51.8 $\pm$ 3.0	37.4 $\pm$ 2.8	25.7 $\pm$ 2.4	40.5

Table 2: Best-of-

N

selection accuracy. BBF : black-box friendly. For non-RCS methods, underline marks the best baseline per setting. Among RCS variants, bold indicates the best RCS method per column. Item-level McNemar tests (Appendix Table 9 ) show that RCS

{}_{\text{uni}}

significantly outperforms self-consistency in 15 of 25 model

\times

benchmark cells (

p<0.05

). Additional results in Appendix Table 8 .

Figure 2: Average performance over five benchmarks for different numbers of sampling responses

N

. Each subplot corresponds to an LLM backbone, showing how accuracy changes with increasing

N

Model	Method	N	Arithmetics				Form.Log.
Model	Method	N	R=0	R=1	R=2	Avg.	R=0	R=1	R=2	Avg.
Qwen2.5-3B	SC	10	94.5 $\pm$ 4.0	60.4 $\pm$ 1.6	34.6 $\pm$ 5.8	63.1	44.8 $\pm$ 2.5	35.3 $\pm$ 2.9	27.6 $\pm$ 5.4	35.9
	RCS ${}_{\text{uni}}$	10	94.3 $\pm$ 3.7	59.5 $\pm$ 4.1	34.0 $\pm$ 11.0	62.6	46.0 $\pm$ 2.3	35.8 $\pm$ 5.3	28.8 $\pm$ 6.0	36.9
	RCS ${}_{\text{freq}}$	10	95.2 $\pm$ 3.6	61.4 $\pm$ 1.2	34.7 $\pm$ 5.4	63.8	44.8 $\pm$ 2.8	36.1 $\pm$ 3.3	28.7 $\pm$ 5.5	36.5
	SC	20	96.5 $\pm$ 0.5	24.8 $\pm$ 3.0	34.0 $\pm$ 3.3	51.8	42.3 $\pm$ 0.5	40.5 $\pm$ 1.6	35.7 $\pm$ 1.6	39.5
	RCS ${}_{\text{uni}}$	20	97.7 $\pm$ 1.0	27.3 $\pm$ 3.2	34.5 $\pm$ 5.1	53.2	46.6 $\pm$ 0.9	45.2 $\pm$ 1.4	37.8 $\pm$ 0.9	43.2

Table 3: Performance on Arithmetics and Form.Log. with Multi-agent Debate. Best values are bolded. R=0,1,2 indicate debate rounds, while Avg. denotes the average over all rounds.

(a) Method GPQA MMLU-Pro BBH-Date BBH-Navigate Avg. SC 13.1 47.6 76.8 39.2 44.2 SCW 12.6 47.6 76.2 39.7 44.0 RCS

{}_{\text{medoid}}

14.6 48.6 76.4 39.2 44.7 RCS

{}_{\text{uni}}

14.1 48.6 78.0 40.8 45.4 RCS

{}_{\text{freq}}

13.6 47.6 77.6 41.2 45.0 RCS

{}_{\text{cosine}}

13.6 47.6 77.2 40.5 44.7

(b) Method Complexity NLL, ANLL

\mathcal{O}(NL)

\mathcal{O}(N)

SCW

\mathcal{O}(N^{2}d)

\mathcal{O}(NL|V|)

ModeX

\mathcal{O}(N^{2}L+N^{3})

RCS

{}_{\text{medoid/cosine}}

\mathcal{O}(N^{2}d)

RCS

{}_{\text{uni/freq}}

\mathcal{O}(Nd)

RCS

{}_{\text{prob}}

\mathcal{O}(N(d+L))

Table 4: Performance under (a) black-box setting (

N{=}

10) and (b) complexity comparison.

Figure 4: Effect of encoders on Llama3.2-3B. Results for Qwen2.5-3B are in Appendix Figure 7 .

Figure 5: Effect of varying

\tau

using Qwen2.5-3B. Results for Llama3.2-3B are in Appendix Figure 8 .

Figure 6: Effect of reasoning paths on Arithmetics and Formal Logic.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Temperature	1
Max new tokens (short-form QA)	32
Max new tokens (long-form tasks)	512

Appendix

Table 5: Sampling hyperparameters used for generation.

Tasks	Prompt
SciQ, GPQA	This is a bot that correctly answers questions.
Arithmetics, GSM8K	Make sure to state your final answer in curly brackets at the very end of your response, just like: {final answer: 12.34} . Let’s think step by step.
Form.Log., MMLU-Pro, BBH-Date	Make sure to state your final answer choice in curly brackets at the very end of your response, just like: {final answer: (A)} . Let’s think step by step.
BBH-Navigate	Make sure to state your final answer choice in curly brackets at the very end of your response, just like: {final answer: Yes} . Let’s think step by step.

Appendix

Table 6: System prompts used for generation across tasks.

Dataset	#Samples	Question	Answer
SciQ	1000	Compounds that are capable of accepting electrons, such as o 2 or f2, are called what?	oxidants
GPQA	198	Which of the following physical theories never requires UV regularization?	Superstring Theory
Arithmetics	200	What is $23+47$ ?	70
GSM8K	200	A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?	3
Formal Logic	126	Select the best translation into predicate logic: Sheena is a punk rocker. Choices: ("Sx", "xS", "sP", "Ps")	"Ps"
MMLU-Pro	105	Determine the number of men needed to build a boat in 77 days if it takes 36 men 132 days to build one. Choices: ("84 men", "36 men", "99 men", "132 men", "45 men", "70 men", "62 men", "50 men", "77 men", "120 men" )	"62 men"

Appendix

Table 7: Overview of evaluation benchmarks, including the number of samples and representative examples.

Model	Method	BBF	SciQ	GPQA	Arithmetics	GSM8K	Form.Log.	Average
Qwen2.5-7B	Greedy	✓	64.1	20.5	97.0	88.3	43.6	63.1
	Oracle	✓	82.0 $\pm$ 1.1	49.9 $\pm$ 1.6	100.0 $\pm$ 0.0	94.6 $\pm$ 0.8	67.7 $\pm$ 2.5	78.8
	NLL	✗	66.4 $\pm$ 0.1	18.3 $\pm$ 1.3	47.3 $\pm$ 5.7	32.7 $\pm$ 4.3	25.7 $\pm$ 0.5	38.1
	ANLL	✗	66.2 $\pm$ 0.4	18.1 $\pm$ 0.9	47.0 $\pm$ 5.6	32.7 $\pm$ 4.4	25.9 $\pm$ 0.5	38.0
	SC	✓	70.3 $\pm$ 0.9	22.1 $\pm$ 4.2	74.3 $\pm$ 2.0	52.3 $\pm$ 2.0	48.4 $\pm$ 0.0	53.5
	SCW	✓	70.2 $\pm$ 0.8	26.3 $\pm$ 3.2	52.8 $\pm$ 5.0	36.4 $\pm$ 5.2	32.8 $\pm$ 3.3	43.7

Appendix

Table 8: Extended Results: Best-of-

N

selection accuracy across settings. Best-of-

N

selection accuracy. BBF : black-box friendly. For non-RCS methods, underline marks the best baseline per setting. Among RCS variants, bold indicates the best RCS method per column. No baseline significantly outperforms RCS at the column level (

p<0.05

, paired t-test); a full per-cell (model

\times

benchmark) McNemar breakdown against all six baselines is in Appendix Table 9 .

Model	Baseline	SciQ	GPQA	Arithmetics	GSM8K	Form.Log.
Qwen2.5-3B	NLL	$<10^{-20}$	$<10^{-4}$	$<10^{-74}$	$<10^{-54}$	$<10^{-16}$
	ANLL	$<10^{-21}$	$<10^{-4}$	$<10^{-74}$	$<10^{-54}$	$<10^{-17}$
	SC	0.001	0.326	$<10^{-7}$	$<10^{-18}$	0.003
	SCW	0.001	1.000	0.108	$<10^{-6}$	0.388
	CE	$<10^{-4}$	$<10^{-4}$	$<10^{-4}$	$<10^{-19}$	0.036
	ModeX	$<10^{-9}$	0.007	$<10^{-39}$	$<10^{-28}$	$<10^{-6}$

Appendix

Table 9: Full per-cell (model

\times

benchmark) item-level McNemar

p

-value of RCS

{}_{\text{uni}}

against each of the six baselines. Bold = significant at

\alpha{=}0.05

in RCS

{}_{\text{uni}}

’s favor; red bold = significant in the baseline’s favor.

Figure 7: Effect of the sentence embedding model on Arithmetics and Form.Log. using Qwen2.5-3B.

Figure 8: Performance when varying correctness threshold (

\tau

) using Llama3.2-3B.

Model	Method	SciQ	GPQA	Arithmetics	GSM8K	Form.Log.
Qwen2.5-3B	NLL	61.7 $\pm$ 1.2	17.6 $\pm$ 3.1	49.2 $\pm$ 12.4	45.2 $\pm$ 4.7	28.0 $\pm$ 3.2
	ANLL	61.6 $\pm$ 1.1	17.1 $\pm$ 4.1	48.7 $\pm$ 11.4	46.0 $\pm$ 4.8	27.5 $\pm$ 3.3
	SC	63.8 $\pm$ 0.5	18.1 $\pm$ 2.7	78.7 $\pm$ 7.2	65.0 $\pm$ 0.5	39.4 $\pm$ 1.8
	SCW	64.0 $\pm$ 0.2	24.0 $\pm$ 2.6	46.8 $\pm$ 1.1	38.1 $\pm$ 6.2	27.0 $\pm$ 2.1
	CE	63.6 $\pm$ 0.3	15.7 $\pm$ 1.7	78.5 $\pm$ 7.4	66.8 $\pm$ 5.0	42.4 $\pm$ 0.2
	ModeX	61.2 $\pm$ 1.2	18.0 $\pm$ 2.4	46.0 $\pm$ 3.0	32.0 $\pm$ 0.5	24.9 $\pm$ 0.9

Appendix

Table 10: Best-of-

N

selection accuracy across settings (

N{=}5

Model	Method	SciQ	GPQA	Arithmetics	GSM8K	Form.Log.
Qwen2.5-3B	NLL	59.0 $\pm$ 0.4	16.4 $\pm$ 2.3	48.2 $\pm$ 11.9	40.6 $\pm$ 0.5	26.5 $\pm$ 2.6
	ANLL	58.4 $\pm$ 0.4	16.2 $\pm$ 2.6	47.8 $\pm$ 12.5	40.8 $\pm$ 1.1	26.5 $\pm$ 2.6
	SC	64.9 $\pm$ 0.6	20.0 $\pm$ 0.3	95.3 $\pm$ 5.1	88.3 $\pm$ 0.9	48.1 $\pm$ 3.6
	SCW	64.9 $\pm$ 0.6	24.4 $\pm$ 2.7	56.2 $\pm$ 4.6	33.8 $\pm$ 3.9	28.3 $\pm$ 1.8
	CE	64.2 $\pm$ 0.9	18.3 $\pm$ 1.3	95.3 $\pm$ 5.9	88.8 $\pm$ 0.9	47.1 $\pm$ 2.4
	ModeX	62.2 $\pm$ 0.8	21.9 $\pm$ 1.7	51.7 $\pm$ 3.4	37.9 $\pm$ 2.1	23.3 $\pm$ 1.8

Appendix

Table 11: Best-of-

N

selection accuracy across settings (

N{=}20

Model	Method	SciQ	GPQA	Arithmetics	GSM8K	Form.Log.
Qwen2.5-3B	NLL	56.3 $\pm$ 0.8	16.8 $\pm$ 1.8	45.3 $\pm$ 13.0	42.8 $\pm$ 1.5	25.9 $\pm$ 3.8
	ANLL	56.1 $\pm$ 0.9	16.8 $\pm$ 2.0	44.8 $\pm$ 13.3	43.0 $\pm$ 2.8	25.1 $\pm$ 3.8
	SC	65.2 $\pm$ 0.4	21.8 $\pm$ 1.4	97.3 $\pm$ 2.9	90.2 $\pm$ 1.1	49.5 $\pm$ 1.2
	SCW	65.3 $\pm$ 0.3	25.6 $\pm$ 2.3	54.5 $\pm$ 4.2	38.1 $\pm$ 1.8	30.7 $\pm$ 3.0
	CE	65.2 $\pm$ 0.3	20.4 $\pm$ 2.7	97.5 $\pm$ 2.6	90.4 $\pm$ 0.5	48.9 $\pm$ 3.2
	ModeX	63.0 $\pm$ 1.0	20.0 $\pm$ 0.5	49.8 $\pm$ 3.2	35.7 $\pm$ 3.1	25.7 $\pm$ 6.7

Appendix

Table 12: Best-of-

N

selection accuracy across settings (

N{=}40

Best-of-N Sampling

Best-of-N sampling generates multiple candidate outputs and selects one using reward, confidence, agreement, or task-specific evidence.

See all 33 papers

Jun 3, 2026

Boosting Self-Consistency with Ranking

Self-consistency improves large language models by sampling multiple reasoning paths and selecting the most frequent answer, but majority voting often fails to recover correct answers that are already present among the samples. We address this limitation with Ranking-Improved Self-Consistency (RISC), which reformulates answer selection in self-consistency as a ranking problem. Instead of relying on a single uncertainty or confidence signal, RISC uses a lightweight LambdaRank model to score candidate answers with five carefully designed features that capture answer frequency, semantic centrality, and reasoning-trace consistency. We evaluate RISC on three datasets under a range of test-time budgets. Across datasets, RISC consistently achieves a better accuracy-efficiency trade-off than standard self-consistency and strong baselines, with particularly large gains on question answering benchmarks. Further analysis shows that the proposed features are individually useful and, more importantly, complementary, highlighting the value of learning to combine multiple informative signals for test-time answer selection.

Maria Marina, Daniil Moskovskiy, Sergey Pletenev +3

AIRI · Skoltech · EPFL

Self-Consistency Decoding LLM Reranking Reasoning Consistency in Language Models Self-Consistency in Language Models

Ask PDF

Mar 12, 2026

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.

Eddie Landesberg

LLM Evaluation LLM-as-a-Judge Best-of-N Sampling LLM Reranking

Ask PDF

Jul 11, 2026

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

Selecting the correct answer from a pool of candidate reasoning chains is the engine of test-time scaling, yet the standard selectors each carry a cost: self-consistency inherits the errors of the single model it resamples, and trained reward models need labeled data and transfer poorly off-distribution. We study a third signal, free at inference time: cross-model consensus, the degree to which independently trained models, each solving the problem once, agree on a final answer. We treat the panel as an LLM-jury, in which the structure of agreement, not any model's score of another, is the verification signal. Across seven benchmarks it selects correct answers better than self-consistency and far better than a model scoring its own candidates: on competition math it closes the entire gap to an oracle selector, while self-scoring closes almost none. The mechanism is error decorrelation: independently trained models err differently, so their wrong answers scatter while the correct one accumulates agreement. We make this precise with a parameter-free law, derived in closed form, that predicts consensus accuracy from three measured panel statistics to a mean absolute error of

0.03

and exposes the method's ceiling: a shared-error floor where models share a misconception, near zero on math but non-trivial on science. Against four trained verifiers spanning discriminative, outcome, and generative reward models, the free LLM-jury matches the strongest inside their math training domain and is the top selector outside it. Cross-model consensus is thus a verifier we can characterize in advance: a law that says when to trust it, and a floor that marks where it cannot.

Ning Liu

Independent Researcher

LLM Answer Verification Test-Time Scaling Multi-Agent Consensus Language Model Ensembles LLM Reasoning

Ask PDF

Abstract

Best-of-N Sampling

Explore similar work

Boosting Self-Consistency with Ranking

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning