Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

Markus Frohmann; Mahdiyar Ali Akbar Alavi; Elizabeth Lingg; Navid Rekabsaz

cs.CLAug 27, 2026

Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

Authors: Markus Frohmann, Mahdiyar Ali Akbar Alavi, Elizabeth Lingg, Navid Rekabsaz

Organizations: Thomson Reuters Labs · University of Toronto, Vector Institute

Abstract

In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or which chosen/rejected pair enters preference training. Because the candidates share that prompt, reordering them changes their scores. The same query over the same candidates should still yield the same decision. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. No prompt-time change we test resolves that dependence: the only one that improves ranking quality does not measurably improve decision stability. We introduce order-consistency SFT (OC-SFT), which attenuates it in the weights by penalizing disagreement between a candidate's scores across orderings. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks. It is also more stable on 12 base models than order-averaged distillation, which trains on labels averaged across permutations. One OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison of such scorers should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.

Figures & tables

Panel of Figure 1 — Figure 1: The same candidates reordered give the same nDCG@10 but a different decision. (a) Each chip is one document, in the same position in both rows rather than by rank: solid where retained, dotted where not, outlined where they disagree. (b) Five trained scorers; values in Table 1 .

Panel of Figure 2 — Figure 2: Order dependence can be addressed at serving, in the labels, or once in training. Each row is one approach, and the filled marker is where it encounters several permutations; single-order distillation, the main baseline, never sees more than one. $T$ counts permutations of one training instance and $K$ permutations of one query; the two are independent, and OC-SFT’s two views are the $N{=}2$ of its objective. We use $T{=}10$ and $K{=}10$ ; Figure 4 measures how far averaging gets.

Untrained
	Passage reranking (18)			Multi-doc QA (3)			Response ranking (5)
Variant	nDCG@10 $\uparrow$	$\tau$ -PSI $\downarrow$	Jacc. $\uparrow$	nDCG@10 $\uparrow$	$\tau$ -PSI $\downarrow$	Ans. $\downarrow$	nDCG@1 $\uparrow$	$\tau$ -PSI $\downarrow$	Pair $\downarrow$
Off the shelf	0.370	0.298	0.439	0.911	0.224	0.221	0.655	0.345	0.877
CapCal	0.372	0.293	0.427	0.911	0.222	0.217	0.657	0.338	0.874
Round-robin	0.422	0.297	0.443	0.911	0.224	0.221	0.658	0.347	0.878
BSC ( $\times$ 10) a	0.465	0.180	0.707	0.946	0.143	0.157	0.716	0.184	0.636

Table 1: Quality saturates on all three tasks; what the consumers decide does not. Each task block gives quality, order instability, and its consumer’s decision: retained-set overlap (Jacc.), answer flip and chosen/rejected-pair flip. Bold marks a Holm-adjusted lead over the other trained variants on all three seeds ( Section B.4 ); no quality column qualifies ( Section 6.1 ). Basis: Qwen3-4B,

M{=}10

random permutations, trained rows self-distilled 3-seed means; BSC is batched self-consistency ( Korikov et al., 2025 ) over the off-the-shelf scorer. Collections and per-task widths are in App. B .

Panel of Figure 3 — Figure 3: Ranking quality moves across permutations and barely with the method; quality and retained-set overlap separate in every family. (a) Qwen3-4B per-query nDCG@10 over $M{=}10$ random permutations on 18 collections; spread is mean minus worst. (b) Dots are 198 base-collection cells over 11 dense bases; overlap at a per-collection F1-tuned cutoff, pooled is their mean.

Panel of Figure 4 — Figure 4: OC-SFT at $K{=}1$ beats BSC at $K{=}10$ permutations on two of the three consumers. Batched self-consistency (BSC, grey; Korikov et al., 2025 ) reaches OC-SFT’s own $K{=}1$ (dashed) only on (c), near $K{=}8$ ; averaging keeps improving OC-SFT itself. BSC’s $K{=}10$ is its Table 1 row.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

	Quality, one permutation		Order instability ( $M=10$ )
Choice	nDCG@10 $\uparrow$	Wins	$\tau$ -PSI $\downarrow$	Lower	Higher
Score readout , quality only
argmax grade	0.3365	1 / 18	—	—	—
$P(g\geq 2)$	0.3701	9 / 18	—	—	—
expected grade	0.3694	8 / 18	—	—	—
Answer skeleton placeholder

Appendix

Table 2: Score-readout and answer-skeleton placeholder choices. Basis: off-the-shelf Qwen3-4B over the 18 reranking collections. Quality is one-permutation nDCG@10; instability is

\tau

-PSI at

B=20

over

M=10

random permutations. Higher quality and lower instability are better; the final two columns count collections with lower / higher instability than Grade: 0 . Bold marks the selected protocol choice rather than the per-column optimum.

Task / group	Collection	Domain / role	Queries / prompts	Candidate pool / protocol
Reranking: in-domain	DL19	web passage; TREC DL	43	BM25 top 100; graded judgments
	DL20	web passage; TREC DL	54	BM25 top 100; graded judgments
	DL21	web passage; TREC DL v2	53	BM25 top 100; graded judgments
	DL22	web passage; TREC DL v2	76	BM25 top 100; graded judgments
	DL23	web passage; TREC DL v2	82	BM25 top 100; graded judgments
Reranking: public OOD	NFCorpus	biomedical	308	BM25 top 100

Appendix

Table 3: Evaluation collections across the three tasks. Counts are evaluated queries or prompts after filtering; protocols are in Section 5 .

	Passage reranking	Multi-document QA	Response ranking
Training data	MS MARCO, $\sim$ 30K queries and $\sim$ 3M document labels	HotpotQA distractor, 30K questions	UltraFeedback, $\sim$ 30.7K prompts
Training width	$B=20$	$B=10$	$B=4$
Rubric	graded $0$ – $3$ topical relevance	answer support, binary	response quality
Consistency weight	$\lambda=5$	$\lambda=3$	$\lambda=1$
Selection split	held-out MS MARCO	disjoint 500 questions	held-out UltraFeedback
Quality metric	nDCG@10	nDCG@10	nDCG@1

Appendix

Table 4: What differs between the three tasks. Every other setting is shared.

Panel of Figure 5 — Figure 5: Residual variation consistent with companion coupling dominates off-the-shelf score variance; OC-SFT removes nearly all of it and flattens the slot residual. Basis: Qwen3-4B; (a–b) use eleven and four collections, while (c–d) use a separate held-out MS MARCO sample of 200 queries at 500 permutations.

Quantity	Off the shelf	Single-order	Order-avg.	OC-SFT
Pool-PSI	0.102	0.099	0.063	0.031
Top-10 changed	74.6%	65.0%	51.5%	36.3%
Mean $\Delta$ nDCG@10	$-0.001$	$+0.001$	$+0.001$	$+0.001$
Mean absolute $\Delta$ nDCG@10	0.0356	0.0240	0.0153	0.0113

Appendix

Table 5: Pool perturbations change decisions without changing mean quality. Lower is better except for signed mean

\Delta

nDCG@10. Basis: Qwen3-4B, seed 42, 15 reranking collections, up to 100 judged queries each; rankings are restricted to the 99 documents shared by the two pools; the sampled rank, dropped document and replacement document are identical across the four columns for every query. DL21–23 are excluded because their first-stage files stop at rank 100 and do not contain the replacement candidate at rank 101.

(a) Ranking-level slot preference
Diagnostic	Collections with primacy	Collections with U-shape
Forced-position nDCG@10	0/18	0/18

Appendix

Table 6: Three controls separate order mechanisms and perturbation transfer. Block (a) tests whether the slot residual reaches ranking quality; block (b) tests transfer to marker renaming; block (c) tests the first-stage-order deficit. Basis: blocks (a) and (c) use Qwen3-4B on the 18 reranking collections; block (b) uses Qwen3-1.7B and Qwen3-4B on DL19, NFCorpus, and Legal-A , with Roman-numeral and random three-character marker schemes held out from training.

(a) Quality controls
Diagnostic	Collection	Single-order	Order-avg.	OC-SFT
$\|\mathrm{corr}(\Delta\mathrm{nDCG@10},\text{ off-the-shelf }\tau\text{-PSI})\|$	DL19	0.257	0.194	0.196
	DL20	0.070	0.037	0.106
Queries improving over off the shelf	DL19	93.0%	97.7%	90.7%
	DL20	90.7%	90.7%	88.9%
Regressions relative to single-order	DL19	reference	37.2%	27.9%

Appendix

Table 7: Validity controls for the stability result. Block (a) separates stability from quality gain; block (b) tests score collapse and tied rankings; block (c) tests movement at the ranking head. Blocks (b–c) use the 18 reranking collections. Basis: Qwen3-4B, seed 42,

M=10

random permutations for blocks (b–c), and the document-ID tie-break.

Variant	mean-perm nDCG	worst-perm nDCG	order-spread	$\tau$ -PSI
Off the shelf	0.4208	0.2940	0.1268	0.2874
Single-order	0.4606	0.3736	0.0870	0.2245
Order-avg.	0.4726	0.4076	0.0650	0.1388
OC-SFT	0.4771	0.4296	0.0475	0.0858

Appendix

Table 8: The variant ordering holds across metrics. Columns report mean and worst nDCG, order-spread and

\tau

-PSI for four variants; higher nDCG and lower instability are better. Basis: eleven dense bases by the 18 reranking collections and

M=10

random permutations; nDCG uses the order-independent document-ID tie-break defined in Section B.1 ; order-spread is mean minus worst before rounding.

(a) Training gain under fixed partitioning
Partition rule	Off the shelf	OC-SFT	Training gain [95% CI]
Contiguous	0.370	0.459	$+0.089$ [ $+0.069$ , $+0.110$ ]
Round-robin	0.422	0.474	$+0.052$ [ $+0.034$ , $+0.073$ ]
Contiguous minus round-robin	$-0.052$	$-0.016$	$+0.036$ [ $+0.022$ , $+0.050$ ]

Appendix

Table 9: Round-robin partitioning does not explain the trained gain. Block (a) gives first-stage-order single-pass nDCG@10 under each partition rule; training gain is OC-SFT minus off the shelf. Block (b) gives round-robin minus contiguous

\tau

-PSI over

M=10

random permutations. Block (c) gives round-robin minus contiguous consumer output for the off-the-shelf scorer; every interval covers zero. Basis: Qwen3-4B and 18 reranking collections; blocks (a–b) use seed 42 and equal-weight collection means, while block (c) uses

M=10

permutations under each rule and the two claim-verification collections for verdict flip. Block (a) intervals, and block (c)’s retained-set-overlap and selected-pair rows, are paired collection bootstraps with 20,000 resamples; the two verdict-flip rows are paired query bootstraps within SciFact and Climate-FEVER.

Variant	Mean nDCG@10	Mean $\tau$ -PSI	% closed
Single-order ( $T{=}1$ )	$0.449\pm 0.003$	$0.209\pm 0.008$	0%
+ LoRA dropout 0.20	$0.448\pm 0.004$	$0.208\pm 0.001$	$+0.8\%$
+ KL-to-base $\beta=0.05$	$0.447\pm 0.002$	$0.231\pm 0.012$	$-17.5\%$
+ EMA mean-teacher $\alpha=0.5$	$0.453\pm 0.003$	$0.119\pm 0.003$	71.4%
Order-avg. $T{=}2$	$0.451\pm 0.002$	$0.176\pm 0.008$	26.2%
Order-avg. $T{=}3$	$0.451\pm 0.000$	$0.153\pm 0.002$	44.4%

Appendix

Table 10: Mitigation baselines and regularizers on the Qwen3-4B student. Higher nDCG@10 and lower

\tau

-PSI are better. Each variant is the mean

\pm

sample standard deviation. Bold is a descriptive seed-spread heuristic, not the Holm-adjusted marking of Table 1 : a lead larger than the seed spread of the rows it leads. The

\tau

-PSI column clears that heuristic, whereas the quality column does not, so quality is unmarked. % closed is the fraction of the stability difference between single-order distillation and OC-SFT that the method removes.

T

is the number of shuffled teacher orders averaged into each label; the

T{=}2

3

and

5

draws are the first

T

orders of the

T{=}10

pool rather than independent draws. Basis: Qwen3-4B, 18 reranking collection means, 3 training seeds.

Student training width	$\Delta$ nDCG@10	95% CI	Wins (of 18)
$B_{\mathrm{train}}=1$	$+0.0291$	[ $+0.0195$ , $+0.0385$ ]	17
$B_{\mathrm{train}}=20$	$+0.0097$	[ $+0.0023$ , $+0.0169$ ]	13

Appendix

Table 11: Batched-teacher labels improve student nDCG@10 at both training widths. The upper block reports the difference between students trained on batched- and pointwise-teacher labels; positive values favor batched-teacher labels. All students are evaluated at serving width

B=1

in first-stage order. Rank-quantile matching assigns the batched teacher’s sorted score values to documents in the pointwise teacher’s rank order. The residual is the batched-label student’s nDCG@10 minus that of the recalibrated pointwise-label student; minimum and maximum are over three evaluated checkpoints. Basis: mean over the 18 reranking collections; Qwen3-4B; training seed 42; 100,000 paired collection-bootstrap samples.

Stage varied	Sweep	Metric	Observed range / change	Forward-pass cost
Student training	$N\in\{2,3,4,8\}$	nDCG@10	range 0.4786–0.4795	100–400% training
Student training	$N\in\{2,3,4,8\}$	$\tau$ -PSI	range 0.067–0.077	100–400% training
Averaged silver, plain SFT	$T=5\rightarrow 40$	nDCG@10	change $+0.0042$	800% labeling
Averaged silver, plain SFT	$T=5\rightarrow 40$	$\tau$ -PSI	change $-0.0398$	800% labeling

Appendix

Table 12: Extra student views raise training cost; extra teacher passes mainly lower

\tau

-PSI.

N

is the number of shuffled student views per training step;

T

is the number of teacher permutations averaged into each stored label. Student-view rows report the range over

N\in\{2,3,4,8\}

; teacher-label rows report the change from

T{=}5

T{=}40

. Cost is relative to

N{=}2

training for student-view rows and

T{=}5

labeling for teacher-label rows. Basis: Qwen3-4B at the selected operating point; student-view rows are a five-collection mean over DL19, DL20, Touché-2020, NFCorpus and Legal-A at a single seed, not the 18-collection default, so they are not comparable to the means in Table 1 .

Base	Round	$\Delta$ nDCG@10	$\Delta\tau$ -PSI
Qwen3-4B	1	$+0.0069$	$-0.0183$
Qwen3-4B	2	$+0.0011$	$+0.0011$
Gemma-E4B	1	$+0.0141$	$-0.0311$
Gemma-E4B	2	$+0.0036$	$-0.0077$

Appendix

Table 13: Iterative self-distillation gains diminish after the first round. Positive

\Delta

nDCG@10 and negative

\Delta\tau

-PSI are improvements. Round 1 is measured against the initial trained student and round 2 against round 1. Basis: 18 reranking collections, one iterative training lineage per base.

Figure 6: OC-SFT at

K=1

is within 0.006 nDCG@10 of the off-the-shelf

K=10

ensemble. Left: nDCG@10 against self-consistency permutations

K

, higher is better. Right: single-pass

\tau

-PSI at

B=20

for off the shelf, single-order distillation, order-averaged distillation and OC-SFT, evaluated over

M=10

random permutations; lower is better.

Panel of Figure 7 — Figure 7: Stability saturates with fewer labeled queries than quality. (a) Student $\tau$ -PSI, lower is better; (b) nDCG@10, higher is better, both against the number of unique labeled training queries over nested pools. Basis: Qwen3-4B; 16 public reranking collections; Legal-A and Legal-B follow the same pattern.

	Off the shelf			Single-order			OC-SFT
Collection	Gold cross. $\downarrow$	1 perm.	10 perm.	Gold cross. $\downarrow$	1 perm.	10 perm.	Gold cross. $\downarrow$	1 perm.	10 perm.
RewardBench-2	0.661	0.594	0.711	0.656	0.594	0.684	0.295	0.683	0.705
Nectar	0.714	0.225	0.295	0.698	0.261	0.300	0.597	0.276	0.318
PPE-MATH	0.654	0.754	0.834	0.523	0.806	0.846	0.426	0.812	0.843
PPE MMLU-Pro	0.727	0.662	0.691	0.770	0.663	0.668	0.593	0.667	0.677
RM-Bench	0.576	0.709	0.711	0.564	0.707	0.725	0.496	0.712	0.722

Appendix

Table 14: Gold crossing and label accuracy on response ranking. Gold crossing is the share of queries whose top-scored response agrees with gold under some permutations and not others; 1 perm. and 10 perm. are label accuracy from a single permutation and from the ten-permutation average. Lower crossing and higher accuracy are better. The binary-gold mean excludes Nectar, whose gold is graded and whose off-the-shelf accuracy is near chance. Basis:

M=10

random permutations; off the shelf is a fixed evaluation, single-order distillation uses seed 42, and OC-SFT is the mean over seeds 42–44.

	Off the shelf		Single-order		Order-avg.		OC-SFT
First stage	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI
BM25	0.374	0.287	0.452	0.198	0.463	0.140	0.463	0.089
BGE (dense)	0.373	0.315	0.462	0.213	0.478	0.147	0.474	0.089
SPLADE (learned-sparse)	0.358	0.307	0.455	0.210	0.463	0.146	0.465	0.090

Appendix

Table 15: Four variants under three first stages. Each variant has separate nDCG@10 and

\tau

-PSI columns, averaged over the same 13 collections. All trained variants use BM25 candidates during training, so the BGE and SPLADE rows test eval-only transfer. DL21–23 and Legal-A/B are excluded because no BGE or SPLADE candidate sets were available for them. Basis: Qwen3-4B, seed 42; BM25-trained checkpoints; per-collection results are in Table 33 .

		Level		Width effect
Task	Metric	$B=1$	Training width	One permutation	Order-averaged
Reranking: 18 collections ( $B=20$ )	nDCG@10	0.459	0.459	$+0.000$	$+0.020$
Multi-doc QA ( $B=10$ ), 3 collections	nDCG@10	0.918	0.961	$+0.043$	$+0.047$
Response ranking ( $B=4$ ), 5 collections
RewardBench-2 (4 candidates)	nDCG@1	0.670	0.690	$+0.020$	$+0.036$
Nectar (7 candidates)	nDCG@1	0.610	0.617	$+0.006$	$+0.026$

Appendix

Table 16: Serving-width effects differ by task and collection. Rows compare OC-SFT at

B=1

and at the training width in parentheses. Width effect is training-width quality minus

B=1

quality. One-permutation effects include the order residual, whereas order-averaged effects average the training-width scores over

K=10

serving permutations. Basis: Qwen3-4B, seeds 42–44.

	OC-SFT support		Reader: OC-SFT		Reader: pointwise		Reader: ten-order labels
Collection	nDCG@10	$\tau$ -PSI	EM	F1	EM	F1	EM	F1
HotpotQA	0.964	0.082	0.621	0.743	0.575	0.684	0.623	0.734
2WikiMultiHopQA	0.975	0.102	0.431	0.546	0.427	0.532	0.398	0.524
MuSiQue	0.944	0.078	0.417	0.531	0.340	0.459	0.381	0.487

Appendix

Table 17: Support ranking transfers zero-shot; reader accuracy depends on scorer context. Support columns report the OC-SFT scorer at

B=10

. Reader columns report the same frozen Granite-4.1-8B reader over the top five passages from OC-SFT, a pointwise scorer, or a

B=1

scorer trained on labels averaged over ten multi-context teacher orders. Higher nDCG@10, EM and F1 are better; lower

\tau

-PSI is better. All metrics are averaged over the same permutations. Basis: Qwen3-4B, 200 questions per collection, seed 42,

M=10

random permutations.

(a) Mean width effect by collection
Collection	One permutation	Order-averaged
HotpotQA	$+0.0345$	$+0.0377$
2WikiMultiHopQA	$+0.0441$	$+0.0496$
MuSiQue	$+0.0518$	$+0.0545$

Appendix

Table 18: QA width effects are positive on all three collections and come from real companion passages. Block (a) gives the mean one-permutation and order-averaged width effects by collection. Block (b) decomposes the one-permutation three-collection mean into informative-alternative, prompt-length/position and answer-skeleton terms. Width effect is

B=10

quality minus

B=1

quality. Basis: Qwen3-4B OC-SFT, three independently trained checkpoints; each served at both widths.

(a) Mean fixed-weight width effects
Collection	One-permutation effect	Order-averaged effect	Pool size
RewardBench-2	$+0.020$	$+0.036$	4
Nectar	$+0.006$	$+0.026$	7
PPE-MATH	$+0.096$	$+0.132$	8
PPE MMLU-Pro	$+0.074$	$+0.076$	8
RM-Bench	$-0.040$	$-0.033$	6

Appendix

Table 19: Response-ranking width and design effects vary by collection and model family. Block (a) reports mean fixed-weight width effects. Block (b) compares nDCG@1 levels for three training/serving designs under one Qwen3-4B evaluation-file readout; it omits Nectar, whose pointwise level is in Table 20 . Block (c) repeats the batched-minus-pointwise comparison on Gemma-E4B with paired prompt-bootstrap intervals. Blocks (a) and (b) estimate different comparisons. Comparing prompts against UltraFeedback found no exact overlap for RewardBench-2 or either PPE collection. Basis: blocks (a–b) use Qwen3-4B and block (c) uses Gemma-E4B.

Collection	$n$	Pointwise	PairRM	Skywork	OC-SFT	Estimate	95% CI	Estimate	95% CI
		nDCG@1				PairRM $-$ pointwise		OC-SFT $-$ PairRM
Nectar	434	0.5975	0.5914	0.6467	0.6152	$-0.0061$	[ $-0.0430$ , $+0.0315$ ]	$+0.0238$	[ $-0.0115$ , $+0.0591$ ]
RewardBench-2	1,763	0.6052	0.4974	0.8174	0.6903	$-0.1078$	[ $-0.1361$ , $-0.0788$ ]	$+0.1929$	[ $+0.1651$ , $+0.2201$ ]
PPE-MATH	512	0.7188	0.4707	0.7656	0.8047	$-0.2480$	[ $-0.3047$ , $-0.1914$ ]	$+0.3340$	[ $+0.2793$ , $+0.3867$ ]
PPE MMLU-Pro	512	0.5918	0.4961	0.6445	0.6660	$-0.0957$	[ $-0.1523$ , $-0.0371$ ]	$+0.1699$	[ $+0.1133$ , $+0.2246$ ]
RM-Bench	1,327	0.7543	0.6353	0.8237	0.7212	$-0.1191$	[ $-0.1492$ , $-0.0889$ ]	$+0.0859$	[ $+0.0543$ , $+0.1176$ ]

Appendix

Table 20: PairRM and Skywork against matched Qwen3-4B controls. Each row gives matched nDCG@1 levels; PairRM contrasts use paired query bootstraps with 20,000 resamples. Basis: seed 42; matched query IDs per collection; UltraFeedback is excluded because it is train-on-test for PairRM.

Base	Gain	95% CI	Improving
Qwen3-1.7B	$+0.369$	[ $+0.298$ , $+0.423$ ]	17/18
Qwen3-4B	$+0.169$	[ $+0.152$ , $+0.186$ ]	18/18
Qwen3-8B	$+0.254$	[ $+0.233$ , $+0.274$ ]	18/18
Qwen3-14B	$+0.248$	[ $+0.210$ , $+0.282$ ]	18/18
Qwen3-32B	$+0.264$	[ $+0.237$ , $+0.292$ ]	18/18
Gemma-E2B	$+0.186$	[ $+0.173$ , $+0.201$ ]	18/18

Appendix

Table 21: Retained-set overlap gain of OC-SFT over single-order distillation by base model. Each gain is an equal-weight collection mean. Single-order distillation is the baseline because the off-the-shelf fixed threshold degenerates to retain-all on TREC-COVID. The 95% CI column gives paired collection-bootstrap intervals; the last column counts positive collection-level gains. Basis: 18 reranking collections at a frozen per-collection F1 threshold, seed 42, independently selected checkpoints on each base.

Collection	GPT-5.4	OC-SFT	Order-avg.	Perm. aug.	DebiasFirst	Single-order	jina-v3
DL19	0.779	0.882	0.827	0.832	0.835	0.732	0.769
DL20	0.782	0.881	0.812	0.827	0.826	0.743	0.721
DL21	0.846	0.899	0.814	0.832	0.832	0.692	0.756
DL22	0.685	0.825	0.785	0.774	0.774	0.638	0.685
DL23	0.617	0.814	0.701	0.747	0.741	0.634	0.621
Touché-2020	0.832	0.832	0.757	0.795	0.773	0.622	0.670

Appendix

Table 22: Retained-set overlap by collection for seven single-permutation systems within 0.022 nDCG@10. The quality and overlap winners differ on 11 of 18 collections. Excluding GPT-5.4, whose

\tau

-PSI is not reported, median pairwise disagreement with overlap is 0.067 for

\tau

-PSI and 0.267 for nDCG@10. The six-system discordance calculation uses seed-42 nDCG@10 and

\tau

-PSI, with jina at matched

B=20

. F1 spread is a query’s mean retained-set F1 minus its worst permutation, averaged over queries, so the set a threshold admits is about as permutation-sensitive as the ranking it is read from. Basis: three-seed means for trained rows and single evaluations for external rows, at the frozen per-collection F1 thresholds of Table 1 ; per-collection quality and instability are in Tables 35 , 36 and 32 .

(a) Repeated-fit flip and accuracy estimates
Contrast	Collection / output	Estimate	SD
Single-order $\to$ OC-SFT	Climate-FEVER	$+0.0263$	0.0031
Single-order $\to$ OC-SFT	SciFact	$+0.0251$	0.0014
Single-order $\to$ OC-SFT, second reader	Climate-FEVER	$+0.0304$	0.0025
Perm. aug. $\to$ EMA	QA exact match	$-0.0228$	0.0029

Appendix

Table 23: Verdict- and answer-flip reductions and controls at 4B. Positive estimates favor the variant after the arrow: lower flip for flip rows and higher accuracy for accuracy rows. Block (a) reports SD where repeated fits are available; block (b) reports query-paired 95% intervals at one fixed fit. In block (c), percentages are the share of queries on which both variants select the same top five. Block (d) uses a query-paired interval for Climate-FEVER and hierarchical query-within-collection intervals for the QA aggregates. Basis: Granite-4.1-8B reader unless Qwen3-4B is named; QA rows use 200 questions per collection.

Collection	Off the shelf	Single-order	Order-avg.	Perm. aug.	DebiasFirst	OC-SFT
RewardBench-2	0.763	0.757	0.496	0.447	0.441	0.356
Nectar	0.905	0.872	0.718	0.704	0.700	0.650
PPE-MATH	0.920	0.914	0.796	0.794	0.792	0.744
PPE MMLU-Pro	0.930	0.940	0.845	0.835	0.837	0.818
RM-Bench	0.869	0.861	0.769	0.776	0.776	0.739

Appendix

Table 24: Preference-pair flip on response ranking. Lower is better. Basis: Qwen3-4B seed 42; Nectar uses the deduplicated 434-prompt subset.

Collection	$n$	Perm. SD	Range	Gap	95% CI
RewardBench-2	1,763	0.0065	0.0199	$-0.136$	[ $-0.154$ , $-0.118$ ]
Nectar	434	0.0162	0.0476	$-0.031$	[ $-0.060$ , $-0.002$ ]
PPE-MATH	512	0.0102	0.0332	$+0.047$	[ $+0.014$ , $+0.080$ ]
PPE MMLU-Pro	512	0.0129	0.0410	$+0.019$	[ $-0.017$ , $+0.055$ ]
RM-Bench	1,327	0.0109	0.0317	$-0.110$	[ $-0.128$ , $-0.092$ ]

Appendix

Table 25: Permutation range exceeds the mean system gap on two collections. Each OC-SFT permutation is aggregated over the prompts scored by pointwise Skywork, so both sides share a basis. Range is max minus min of OC-SFT’s nDCG@1 across

M{=}10

permutations. Bold marks ranges larger than the absolute system gap. Perm. SD is the standard deviation across permutations. Gap is OC-SFT minus Skywork; its intervals are paired prompt bootstraps and carry query sampling rather than permutation range. Basis: Qwen3-4B seed 42; Nectar uses the deduplicated 434-prompt subset.

(a) nDCG@10 width effect
Comparison	Collection basis	Effect	95% CI
One permutation	18 reranking	$+0.0001$	[ $-0.0133$ , $+0.0177$ ]
Order-averaged, original	18 reranking	$+0.0201$	[ $+0.0063$ , $+0.0429$ ]
Order-averaged, matched $B=1$	18 reranking	$+0.0191$	[ $+0.0050$ , $+0.0424$ ]
Order-averaged, original	Excluding ArguAna	$+0.0098$	[ $+0.0051$ , $+0.0147$ ]
Order-averaged, matched $B=1$	Excluding ArguAna	$+0.0088$	[ $+0.0035$ , $+0.0144$ ]

Appendix

Table 26: The

+0.020

passage-reranking gain survives estimator matching, while engine-call batching has little effect on retained-set overlap. Block (a) reports the one-permutation and order-averaged width effects, including a matched

B=1

estimator built from ten independently rebatched passes. Block (b) holds candidate order fixed and varies engine-call batching, then contrasts it with the candidate-order perturbation. Basis: block (a) uses the 18 reranking collections at seed 42; block (b) uses the 16 public ones at seed 42 and the frozen-threshold protocol of App. F .

First-stage order
Collection basis	Width	Content	Length + slot	Skeleton
18 reranking	$-0.0002$	$-0.0072$	$+0.0039$	$+0.0031$
Excluding ArguAna	$-0.0069$	$-0.0047$	$-0.0016$	$-0.0007$
ArguAna	$+0.1144^{\ast}$	$-0.0496^{\ast}$	$+0.0965^{\ast}$	$+0.0675^{\ast}$
Averaged over $K=10$ serving permutations
18 reranking	$+0.0194$	$+0.0085$	$+0.0026$	$+0.0083$

Appendix

Table 27: Decomposition of the width effect. At fixed Qwen3-4B OC-SFT weights,

q_{\mathrm{alone}}

scores a target alone under one answer slot,

q_{\mathrm{comp}}

adds its 19 real companions,

q_{\mathrm{fill}}

substitutes 19 unrelated length-matched fillers, and

q_{\mathrm{skel}}

returns the target alone under the full 20-slot template. Width is

q_{\mathrm{comp}}-q_{\mathrm{alone}}

, content is

q_{\mathrm{comp}}-q_{\mathrm{fill}}

, length plus slot is

q_{\mathrm{fill}}-q_{\mathrm{skel}}

, and skeleton is

q_{\mathrm{skel}}-q_{\mathrm{alone}}

. The upper block uses the first-stage permutation; the lower block averages the

B=20

scores over

K=10

serving permutations. Positive entries mean the wider serving width ranks better. Content, length plus slot, and skeleton sum to width up to rounding. Starred cells have query-paired 95% bootstrap intervals clear of zero. Basis: Qwen3-4B OC-SFT, seed 42, 18 reranking collections.

(a) Fixed-weight width sensitivity, seed 42
Variant	$\tau$ -PSI	Width effect	95% CI
Not trained at the serving width
Off the shelf	0.298	$-0.062$	[ $-0.082$ , $-0.043$ ]
Student trained at $B=1$	0.222	$-0.048$	[ $-0.062$ , $-0.036$ ]
Trained at $B=20$
Order-avg.	0.139	$-0.004$	[ $-0.012$ , $+0.004$ ]

Appendix

Table 28: Training width determines tolerance to serving-width changes. Block (a) measures

\tau

-PSI at

B{=20}

and the single-pass nDCG@10 change from serving the same seed-42 weights at

B{=1}

and

B{=20}

; intervals are paired 95% bootstraps. Block (b) crosses training and serving width using three-seed means; its final column averages ten

B=20

permutations.

B{=1}

\tau

-PSI entries are zero by construction. Basis: Qwen3-4B, 18 reranking collections, equal-weight collection mean.

Variant / contrast	Level or difference	95% CI
Off the shelf	0.521	[0.449, 0.593]
Single-order	0.300	[0.246, 0.383]
Perm. aug.	0.245	[0.198, 0.323]
OC-SFT	0.232	[0.175, 0.330]
Augmentation minus OC-SFT	$+0.013$	[ $-0.012$ , $+0.034$ ]

Appendix

Table 29: Magnitude of the score-level width effect across training variants. Entries are collection-first mean

|\mu_{B}-\mu_{1}|

normalized by pooled across-document score SD; they include companion content and prompt geometry. The final row isolates the consistency penalty. Order-averaged distillation is omitted because it changes the teacher targets. Basis: 18 reranking collections, Qwen3-4B, seed 42;

B=20

averages

K=10

serving permutations,

B=1

scores each document alone, and intervals bootstrap collections.

Serve $B$	1	5	10	20	25	30	34	40	50
18-collection nDCG@10 $\uparrow$	0.459	0.453	0.454	0.459	0.461	0.460	0.446	0.345	0.263
18-collection $\tau$ -PSI $\downarrow$	0	0.094	0.091	0.085	0.083	0.094	0.135	0.233	0.275

Appendix

Table 30: OC-SFT quality is stable through

B=1

B=30

. The selected

B=20

-trained OC-SFT checkpoint is served at nine widths; higher nDCG@10 and lower

\tau

-PSI are better. The training width is bold in the header. Basis: Qwen3-4B, seed 42,

M=10

random permutations for

B>1

; at

B=1

\tau

-PSI is zero by construction.

Figure 8: The system frontier across 16 public reranking collections. The horizontal axis is median per-query latency, the vertical axis mean nDCG@10, color mean

\tau

-PSI, and marker shape whether the system emits a per-document score. The model-family points are trained OC-SFT checkpoints on the named bases. jina, mxbai and RankZephyr are external rerankers shown at their stated serving configurations. Gray denotes

\tau

-PSI zero by construction for the order-free point. nDCG@10 and

\tau

-PSI are equal-weight means and latency is the median of per-collection medians over 16 public reranking collections. Ringed points use four L40S GPUs rather than one.

Collection	$B=1$	$B=10$	$B=20$	$B=10$ $\downarrow$	$B=20$ $\downarrow$
DL19	2.223	1.020	1.086	54.1%	51.2%
DL20	2.237	1.032	1.124	53.9%	49.7%
TREC-COVID	2.214	1.111	1.255	49.8%	43.3%
NFCorpus	2.249	1.092	1.226	51.5%	45.5%
Touché-2020	2.404	1.176	1.315	51.1%	45.3%
DBPedia-Entity	2.235	1.036	1.100	53.6%	50.8%

Appendix

Table 31: Candidate batching roughly halves per-query latency. Collection rows report seconds per query and the reduction relative to

B=1

. Summary rows are equal-collection medians; brackets give 95% bootstrap intervals for the reduction. Basis: Qwen3-4B OC-SFT, seed 42, 18 collections, one L40S, warm vLLM and client concurrency 1. Each collection is timed three times on the same GPU, with the width order varied across repetitions. These within-model timings use query batch size 1 and are separate from the system-level latency comparison in Figure 8 .

nDCG@10, higher is better
Method	Public 16	DL19	DL20	DL21	DL22	DL23	Touché	FiQA	NFCorpus	ArguAna	Climate	T-COVID	DBPedia	SciFact	Signal1M	T-NEWS	Robust04	Legal-A	Legal-B
First-stage retrieval without reranking
BM25 retrieval	0.39	0.51	0.49	0.45	0.27	0.26	0.44	0.24	0.34	0.40	0.17	0.59	0.32	0.68	0.33	0.40	0.41	0.30	0.32
Prompted closed model ( $B=20$ )
GPT-5.4	0.49	0.68	0.65	0.65	0.45	0.42	0.23	0.37	0.38	0.40	0.19	0.85	0.39	0.75	0.27	0.50	0.57	0.36	0.31
Qwen3-4B

Appendix

Table 32: Full per-collection reranking results for the Qwen3-4B student and its baselines. Higher nDCG@10 and lower

\tau

-PSI are better. BM25 has zero

\tau

-PSI by construction; GPT-5.4 has no reported

\tau

-PSI because tied grades leave its relative ordering to the tie-break, which is document identifier for nDCG@10, as noted in Table 1 . Basis: seed 42 and

M=10

random permutations; Public 16 excludes Legal-A and Legal-B, whose candidate lists are precomputed. Body Table 1 reports the mean over all 18 collections.

	BGE (dense)								SPLADE++ ED (learned-sparse)
	Off the shelf		Single-order		Order-avg.		OC-SFT		Off the shelf		Single-order		Order-avg.		OC-SFT
Collection	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI	nDCG	$\tau$ -PSI
DL19	0.573	0.336	0.740	0.202	0.758	0.131	0.740	0.073	0.586	0.333	0.739	0.209	0.727	0.134	0.741	0.075
DL20	0.533	0.328	0.720	0.193	0.729	0.130	0.723	0.074	0.513	0.331	0.719	0.197	0.718	0.129	0.713	0.074
Touché-2020	0.219	0.300	0.262	0.187	0.267	0.113	0.273	0.070	0.193	0.287	0.247	0.178	0.241	0.114	0.246	0.079
FiQA	0.256	0.304	0.360	0.210	0.377	0.131	0.382	0.070	0.250	0.292	0.354	0.202	0.356	0.128	0.365	0.070

Appendix

Table 33: Per-collection results under dense and learned-sparse first stages. Each variant has separate nDCG@10 and

\tau

-PSI columns. The three trained variants use BM25 candidates during training; the BGE and SPLADE results therefore measure transfer to new first stages without retraining. Higher quality and lower instability are better. Basis: Qwen3-4B, seed 42 and first-stage-specific judged sets; the Mean (13) row reproduces the BGE and SPLADE rows in Table 15 , whose first row reports BM25.

nDCG@10, higher is better
Method	Public 16	DL19	DL20	DL21	DL22	DL23	Touché	FiQA	NFCorpus	ArguAna	Climate	T-COVID	DBPedia	SciFact	Signal1M	T-NEWS	Robust04	Legal-A	Legal-B
RankZephyr-7B	0.50	0.74	0.70	0.67	0.51	0.44	0.32	0.34	0.37	0.44	0.22	0.78	0.43	0.74	0.31	0.49	0.52	0.35	0.33
jina-reranker-v3	0.50	0.72	0.70	0.69	0.50	0.45	0.30	0.38	0.37	0.68	0.24	0.74	0.43	0.68	0.31	0.40	0.49	0.35	0.32
jina-reranker-v3 ( $B=20$ )	0.47	0.69	0.65	0.63	0.48	0.42	0.25	0.35	0.34	0.65	0.23	0.73	0.39	0.64	0.25	0.35	0.45	0.27	0.26
mxbai-base-v2	0.50	0.73	0.67	0.66	0.51	0.43	0.35	0.36	0.35	0.59	0.31	0.70	0.44	0.67	0.33	0.42	0.49	0.31	0.27
mxbai-large-v2	0.52	0.73	0.69	0.69	0.50	0.46	0.33	0.39	0.36	0.67	0.35	0.74	0.45	0.73	0.31	0.47	0.51	0.33	0.32

Appendix

Table 34: Purpose-built rerankers and training variants on Qwen3-Reranker-4B. Higher nDCG@10 and lower

\tau

-PSI are better. RankZephyr and jina are listwise rerankers, whereas mxbai is pointwise and has zero

\tau

-PSI by construction. The native jina row uses one context; the matched row uses the students’

B=20

geometry. Purpose-built systems use large hard-negative-mined reranking corpora, whereas OC-SFT uses about 30K MS MARCO queries, so they are external reference systems. Basis: Public 16 excludes Legal-A and Legal-B.

Figure 9: Model size, family and teacher identity. Panels (a) and (b) show mean nDCG@10 and

\tau

-PSI over the 18 reranking collections, respectively, by model size for Qwen3, Gemma and Granite. Gemma E2B and E4B are plotted at their effective sizes, 2B and 4B. Panel (c) compares nDCG@10 across the evaluated Qwen3 student sizes for frontier, open and self-distilled teachers.

Figure 10: Per-variant

\tau

-PSI spread across collections. Panel (a) shows Qwen3-4B across the 18 reranking collections. Panel (b) pools the same collections across all 11 dense bases, for 198 model-collection cells. Diamonds mark means; the shaded region spans the range of model-level mean OC-SFT

\tau

-PSI values. The sparse MoE is excluded from this figure and reported in Table 36 .

Figure 11: Student quality correlates with label order-consistency across teachers. Student nDCG@10 plotted against teacher silver-

\beta

for eight teachers. The correlation with student nDCG@10 is

-0.852

for silver-

\beta

and

+0.812

for silver quality; they correlate at

-0.528

. Shading marks the 95% confidence region for the fitted relationship between silver-

\beta

and student nDCG@10.

Variant	DL19	DL20	DL21	DL22	DL23	Touché-2020	FiQA	NFCorpus	ArguAna	Climate-FEVER	TREC-COVID	DBPedia	SciFact	Signal-1M	TREC-NEWS	Robust04	Legal-A	Legal-B	Public 16	All 18
Qwen3-1.7B
Off the shelf	0.479	0.435	0.452	0.288	0.249	0.208	0.213	0.301	0.318	0.142	0.675	0.258	0.526	0.210	0.307	0.399	0.233	0.209	0.341	0.328
Single-order	0.586	0.547	0.545	0.347	0.317	0.246	0.215	0.289	0.262	0.165	0.703	0.315	0.525	0.232	0.352	0.430	0.180	0.174	0.380	0.357
Order-avg.	0.678	0.615	0.630	0.405	0.367	0.282	0.288	0.336	0.344	0.177	0.736	0.379	0.633	0.296	0.405	0.474	0.287	0.279	0.440	0.423
OC-SFT	0.695	0.633	0.662	0.427	0.391	0.290	0.296	0.347	0.286	0.202	0.728	0.417	0.638	0.292	0.432	0.495	0.305	0.291	0.452	0.435
OC-SFT (avg. labels)	0.678	0.622	0.646	0.421	0.375	0.279	0.301	0.349	0.306	0.190	0.751	0.393	0.657	0.296	0.417	0.477	0.277	0.260	0.447	0.427

Appendix

Table 35: Per-collection nDCG@10 for every base and training variant. Twelve bases, eleven dense and one sparse Mixture-of-Experts, each with its four training variants; higher is better. The final columns report means over the 16 public and all 18 reranking collections. Every block carries a fifth row, OC-SFT trained on order-averaged rather than single-order teacher labels. The first stage is BM25 on the 16 public collections. Legal-A and Legal-B use the precomputed candidate lists packaged with their internal fixtures; the retriever identity is not recorded. Basis: all 18 reranking collections, seed 42, one independently selected checkpoint per base and variant; off-the-shelf rows evaluate fixed base checkpoints. Order instability for the same cells is in Table 36 .

Variant	DL19	DL20	DL21	DL22	DL23	Touché-2020	FiQA	NFCorpus	ArguAna	Climate-FEVER	TREC-COVID	DBPedia	SciFact	Signal-1M	TREC-NEWS	Robust04	Legal-A	Legal-B	All 18
Qwen3-1.7B
Off the shelf	0.393	0.387	0.405	0.404	0.403	0.413	0.415	0.383	0.376	0.342	0.392	0.399	0.377	0.373	0.367	0.365	0.448	0.445	0.394
Single-order	0.329	0.324	0.336	0.352	0.352	0.315	0.361	0.355	0.365	0.388	0.337	0.361	0.407	0.357	0.334	0.370	0.371	0.374	0.355
Order-avg.	0.148	0.148	0.149	0.163	0.163	0.128	0.144	0.179	0.198	0.174	0.137	0.180	0.211	0.181	0.135	0.177	0.172	0.173	0.164
OC-SFT	0.094	0.099	0.084	0.097	0.105	0.074	0.081	0.101	0.153	0.096	0.079	0.104	0.110	0.132	0.087	0.105	0.106	0.101	0.100
OC-SFT (avg. labels)	0.065	0.067	0.062	0.071	0.071	0.057	0.063	0.080	0.114	0.077	0.062	0.085	0.081	0.096	0.064	0.082	0.072	0.071	0.074

Appendix

Table 36: Per-collection

\tau

-PSI for every base and training variant. The same twelve bases and 18 per-collection cells as Table 35 ; lower is more stable. The All 18 column averages all 18 reranking collections. Every block also includes OC-SFT trained on order-averaged labels. Basis: all 18 reranking collections, seed 42,

M=10

random permutations, one independently selected checkpoint per base and variant.

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 130 papers

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation
Apr 30, 2026Ethan Bito, Yongli Ren, Estrid HeLearning to Rank LLM Reranking
RMIT University, Melbourne, Australia
Can LLM Rerankers Predict Their Own Ranking Performance?
Jun 2, 2026Shiyu Ni, Keping Bi, Jiafeng Guo +3Probability Calibration LLM Reranking
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders
Jul 26, 2026Ge Zhang, Jingru Cheng, Huiyuan ChenLearning to Rank Adversarial Attacks
Stanford University · Independent Researcher

Apr 30, 2026J/K move · Enter open · S save

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Ethan Bito, Yongli Ren, Estrid He

RMIT University, Melbourne, Australia

Learning to Rank LLM Reranking Large Language Model-Based Recommendation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for recommendation reranking, but their listwise predictions can depend on the order in which candidates are presented. This creates a mismatch between the set-based nature of recommendation and the sequence-based computation of decoder-only LLMs, where permuting an otherwise identical candidate set can change item scores and final rankings. Such order sensitivity makes LLM-based rerankers difficult to rely on, since rankings may reflect prompt serialization rather than user preference. We propose InvariRank, a permutation-invariant listwise reranking framework that addresses this dependence at the architectural level. InvariRank blocks cross-candidate attention with a structured attention mask and negates position-induced scoring changes through shared positional framing under Rotary Positional Embeddings (RoPE). Combined with a listwise learning-to-rank objective, the model scores all candidates in a single forward pass, avoiding permutation-based invariance training objectives that require multiple permutations of a candidate set. Experiments on recommendation benchmarks show that InvariRank maintains competitive ranking effectiveness while producing stable rankings across candidate permutations. The results suggest that architectural invariance is a practical route to reliable and efficient LLM-based recommendation reranking. The source code is at https://github.com/ejbito/InvariRank.

Abstract

LLM Reranking

Explore similar work

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Can LLM Rerankers Predict Their Own Ranking Performance?

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders