Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence

Carlos Peláez-González; Francisco Herrera; Andrés Herrera-Poyatos

cs.IROct 27, 2026

Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence

Authors: Carlos Peláez-González, Francisco Herrera, Andrés Herrera-Poyatos

Organizations: Department of Computer Science and Artificial Intelligence, Andalusian Institute of Data Science and Computational Intelligence (DaSCI), University of Granada, Spain. · Department of Algebra, University of Granada, Spain.

Abstract

Large Language Models (LLMs) are increasingly used as recommendation generators in conversational recommender systems (RS) via retrieval-augmented generation (RAG). This flexible design introduces new sources of structural bias. This paper studies positional bias in top-1 and single-turn LLM-based RS: systematic dependence of the recommended item on the order in which candidates are presented in the prompt. We propose two complementary detection methodologies: (i) POB-ICM, based on contingency matrix analysis and Chi-square independence testing, and (ii) POB-UNBP, a sample-efficient approach derived from a uniformity null model under permutation-based evaluation, enabling stochastic decoding with formal statistical guarantees. Using a controlled RAG testbed for product recommendation, built on a real product catalog, a fixed prompt template and 2000 template-generated user queries with 20 permutations each (40,000 instances), both methodologies reject the null hypothesis of positional independence and uniformity in all evaluated state-of-the-art models. We introduce three complementary metrics -- permutation sensitivity, positional bias, and strong positional bias -- to quantify this effect, the latter targeting cases where the position displaces the recommendation towards clearly less relevant items rather than only choosing among comparably relevant ones. A robustness analysis across candidate-set size and decoding temperature shows that the bias is structural and model-dependent, not an artifact of deterministic decoding or candidate-set size. These results establish positional bias as a measurable, reproducible property of prompt-mediated recommendation pipelines in the evaluated setting and motivate routine auditing and governance controls (e.g., order randomization, standardized prompting) for deployed LLM-based RS. Source code is publicly released.

Figures & tables

Figure 1: LLM-based RS architecture used in this paper.

Panel of Figure 3 — Figure 3: Titles of the selected products for the query "Find me an eco-friendly protein snack". These are sorted by the cosine similarity to the query using BAAI embedding model.

Model name	Error rate				Refusal rate
Model name	Top-5	Top-10	Top-15	Top-20	Top-5	Top-10	Top-15	Top-20
DeepSeek R1 32B	0.35%	0.39%	0.24%	0.26%	4.03%	3.89%	3.82%	4.06%
DeepSeek R1 7B	0.81%	1.58%	1.99%	2.46%	7.39%	5.38%	5.14%	4.36%
DeepSeek R1 8B	0.37%	0.40%	0.21%	0.23%	0.01%	0.01%	0.00%	0.00%
LLaMA 3.1 8B	0.60%	1.99%	3.23%	4.01%	0.01%	0.01%	0.01%	0.01%
LLaMA 4 Scout	0.21%	0.48%	2.05%	3.20%	0.17%	0.06%	0.08%	0.05%

Table 1: Recommendation parsing error rate and refusal rate for each candidate-set size, each computed over the

40{,}000

samples of the corresponding query-permutation dataset.

Model name	$p$ -value (POB-ICM)	$p$ -value (POB-UNBP)	$L$	Permutation sensitivity (PS)	Positional bias (PB)	Strong Positional Bias (sPB)
DeepSeek R1 32B	<0.0005	7.456e-07	0.97%	21.857	2.563, CI: (2.0, 3.2)	3.1
DeepSeek R1 7B	<0.0005	6.197e-60	4.15%	33.983	5.612, CI: (5.1, 6.3)	9.8
DeepSeek R1 8B	<0.0005	5.875e-05	0.79%	29.758	2.391, CI: (1.8, 3.0)	6.7
LLaMA 3.1 8B	<0.0005	4.457e-26	3.01%	30.395	4.187, CI: (3.6, 4.8)	6.7
LLaMA 4 Scout	<0.0005	1.188e-04	0.87%	22.393	2.886, CI: (2.4, 3.4)	3.2
GPT OSS 120B	<0.0005	8.879e-30	3.14%	15.487	4.368, CI: (3.8, 5.0)	1.0

Table 2: Results from the proposed bias quantification metrics and proposed statistical methodologies for top-

5

setting. POB-ICM

p

-values are Monte Carlo estimates from

B=2000

permutations, so their smallest attainable value is

1/(B+1)=0.0005

L

value is the lower bound of

TV(p(\cdot),U_{k})

according to the POB-UNBP test. Confidence intervals of sPB reported at Appendix E .

Panel of Figure 4 — Figure 4: Contingency matrix from subsection 4.3 , shown as a heatmap. The plot was generated using LLaMA 3.1 8B as recommender. Each column shows the empirical distribution of recommendations conditioned on a permutation, with permutations sorted lexicographically. Red lines indicate when the first, second, third, fourth, and fifth product are shuffled to the first position, respectively. Consistency was ensured by shuffling products according to the permutation sample, generating the recommendation, and then unshuffling the recommendation positions.

Figure 5: Bias analysis given the Equation 3 for different recommender LLMs. For each model, a column of its associated heatmap represents the mean of the (non-normalized) counts of selected products by the model when these were moved to the position represented by said column. In the absence of positional bias, all columns would be identical.

	Top-5		Top-10		Top-15		Top-20
Model name	$p$ -value	$L$	$p$ -value	$L$	$p$ -value	$L$	$p$ -value	$L$
DeepSeek R1 32B	7.456e-07	0.97%	5.130e-17	2.83%	2.202e-47	4.29%	2.098e-48	6.78%
DeepSeek R1 7B	6.197e-60	4.15%	9.362e-55	5.79%	5.966e-79	9.51%	4.449e-70	9.89%
DeepSeek R1 8B	5.875e-05	0.79%	1.150e-11	1.96%	2.253e-46	4.30%	2.111e-71	7.24%
LLaMA 3.1 8B	4.457e-26	3.01%	4.928e-17	3.13%	3.379e-16	4.08%	1.465e-26	3.77%
LLaMA 4 Scout	1.188e-04	0.87%	1.438e-60	8.30%	8.023e-208	21.43%	1.341e-249	30.19%

Table 3: POB-UNBP statistical test

p

-value and the computed lower bound

L

for different top-

K

values and a fixed temperature

t=0

. Bold values represent statistically detectable positional bias according to the threshold of

0.05

	$t=0.0$		$t=0.2$		$t=0.4$		$t=0.6$
Model name	$p$ -value	$L$	$p$ -value	$L$	$p$ -value	$L$	$p$ -value	$L$
DeepSeek R1 32B	7.456e-07	0.97%	2.001e-09	1.14%	6.392e-06	0.79%	6.290e-07	1.12%
DeepSeek R1 7B	6.197e-60	4.15%	1.924e-64	4.29%	7.115e-58	4.24%	2.998e-66	4.40%
DeepSeek R1 8B	5.875e-05	0.79%	8.181e-06	0.90%	9.225e-08	1.21%	1.701e-16	1.65%
LLaMA 3.1 8B	4.457e-26	3.01%	2.049e-28	2.67%	4.884e-21	2.27%	1.925e-17	1.97%
LLaMA 4 Scout	1.188e-04	0.87%	1.476e-05	0.97%	5.367e-05	0.97%	6.510e-05	1.00%

Table 4: POB-UNBP statistical test

p

-value and the lower bound

L

for different decoding

t

temperatures and a fixed

K=5

. Bold values represent statistically detectable positional bias according to the threshold of

0.05

Panel of Figure 6 — Figure 6: Representation of probability distribution p(i) defined in Equation 1 for different $k$ values. Red-dashed lines represent uniform baseline $1/k$ .

Model name	Temperature	p(1)	p(2)	p(3)	p(4)	p(5)
DeepSeek R1 32B	0.0	0.1874	0.1869	0.2060	0.2083	0.2114
	0.2	0.1869	0.1868	0.2046	0.2080	0.2137
	0.4	0.1873	0.1879	0.2055	0.2097	0.2097
	0.6	0.1872	0.1852	0.2053	0.2103	0.2119
DeepSeek R1 7B	0.0	0.1738	0.2514	0.2004	0.2043	0.1701
	0.2	0.1686	0.2535	0.2012	0.2039	0.1729

Table 5: Probability distributions p(i) defined in Equation 1 for different temperature values and top-5 setting. Uniform baseline is

0.2

Figure 7: Selection counts of each candidate item, by original (pre-shuffling) position, for twenty random queries of the exhaustive configuration, using LLaMA 3.1 8B. Each panel shows, for one query, how many of the 120 permutations recommended each item.

Model name	PS				PB				sPB
Model name	5	10	15	20	5	10	15	20	5	10	15	20
DeepSeek R1 32B	21.9	29.4	34.1	37.2	2.6	5.0	7.1	9.4	3.1	6.9	10.7	13.7
DeepSeek R1 7B	34.0	45.1	52.7	58.1	5.6	8.9	13.5	13.0	9.8	20.1	26.6	33.2
DeepSeek R1 8B	29.8	38.6	43.2	46.4	2.4	4.5	7.1	10.1	6.7	14.8	19.4	20.6
LLaMA 3.1 8B	30.4	41.9	46.9	49.6	4.2	5.3	6.8	7.0	6.7	15.7	20.7	24.1
LLaMA 4 Scout	22.4	38.3	51.5	61.6	2.9	10.5	24.3	33.8	3.2	12.9	24.5	36.6

Table 6: Permutation sensitivity (PS), positional bias (PB), and strong positional bias (sPB), all expressed as percentages, across

k\in\{5,10,15,20\}

. Confidence intervals of sPB metric reported in Appendix E .

Model name	PS				PB				sPB
Model name	0.0	0.2	0.4	0.6	0.0	0.2	0.4	0.6	0.0	0.2	0.4	0.6
DeepSeek R1 32B	21.9	22.7	22.9	23.4	2.6	2.6	2.5	2.8	3.1	3.3	3.4	3.5
DeepSeek R1 7B	34.0	34.6	35.3	35.9	5.6	5.9	5.9	6.0	9.8	10.0	10.4	10.4
DeepSeek R1 8B	29.8	30.9	31.5	32.6	2.4	2.1	2.7	2.8	6.7	6.9	8.7	9.9
LLaMA 3.1 8B	30.4	31.4	32.3	33.0	4.2	4.0	3.6	3.0	6.7	7.1	9.3	9.4
LLaMA 4 Scout	22.4	22.6	22.6	22.9	2.9	3.1	3.1	3.0	3.2	3.2	3.2	3.3

Table 7: Permutation sensitivity (PS), positional bias (PB), and strong positional bias (sPB), all expressed as percentages, for top-

5

across decoding temperatures

t\in\{0.0,0.2,0.4,0.6\}

. Confidence intervals of sPB metric reported in Appendix E .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model name	sPB			$L$		IoU of $\mathcal{R}(X)$		$p$ -value
Model name	(i)	(ii)	(iii)	(ii)	(iii)	(i), (ii)	(i), (iii)	$p$ -value
DeepSeek R1 32B	2.9 (2.8, 3.1)	3.1 (2.9, 3.2)	3.4 (3.3, 3.6)	1.0	1.1	78.9	79.8	2.2e-08
DeepSeek R1 7B	9.1 (8.8, 9.4)	9.8 (9.5, 10.1)	10.2 (9.9, 10.5)	4.2	4.4	70.2	70.6	7.4e-09
DeepSeek R1 8B	6.6 (6.3, 6.8)	6.7 (6.5, 6.9)	7.5 (7.3, 7.8)	0.8	1.6	73.6	73.4	4.3e-11
LLaMA 3.1 8B	6.9 (6.7, 7.2)	6.7 (6.4, 6.9)	7.5 (7.3, 7.7)	3.0	2.0	74.3	74.5	1.6e-08
LLaMA 4 Scout	2.3 (2.1, 2.5)	3.2 (3.0, 3.4)	3.3 (3.1, 3.5)	0.9	1.0	76.4	76.6	5.1e-19

Appendix

Table 10: Strong positional bias (sPB), POB-UNBP lower bound

L

\mathrm{TV}(p(\cdot),U_{k})

for the configurations with permutations, and mean intersection over union (IoU) of the relevance sets with configuration (i), for top-

5

under configurations (i) stochastic decoding, (ii) random permutation and (iii) both. sPB and IoU are percentages and

L

is in percentage points, with

95\%

confidence intervals for sPB. The last column reports the Holm-adjusted

p

-value comparing the sPB of configurations (iii) and (i); bold values are significant at the

0.05

level.

Model name	sPB			$L$		IoU of $\mathcal{R}(X)$		$p$ -value
Model name	(i)	(ii)	(iii)	(ii)	(iii)	(i), (ii)	(i), (iii)	$p$ -value
DeepSeek R1 32B	2.6 (2.4, 2.8)	3.1 (2.9, 3.2)	3.5 (3.3, 3.6)	1.0	2.6	78.6	78.9	3.0e-18
DeepSeek R1 7B	9.5 (9.2, 9.8)	9.8 (9.5, 10.1)	10.4 (10.1, 10.7)	4.2	3.9	71.0	71.3	2.4e-07
DeepSeek R1 8B	6.0 (5.8, 6.3)	6.7 (6.5, 6.9)	7.2 (6.9, 7.4)	0.8	3.7	72.4	72.4	1.3e-14
LLaMA 3.1 8B	6.0 (5.7, 6.2)	6.7 (6.4, 6.9)	7.3 (7.0, 7.5)	3.0	0.3	73.9	74.0	7.8e-21
LLaMA 4 Scout	2.7 (2.5, 2.9)	3.2 (3.0, 3.4)	3.2 (3.1, 3.4)	0.9	0.9	79.2	80.6	2.4e-07

Appendix

Table 11: Strong positional bias (sPB), POB-UNBP lower bound

L

\mathrm{TV}(p(\cdot),U_{k})

for the configurations with permutations, and mean intersection over union (IoU) of the relevance sets with configuration (i), for top-

5

under configurations (i) random labels, (ii) random permutation and (iii) both. sPB and IoU are percentages and

L

is in percentage points, with

95\%

confidence intervals for sPB. The last column reports the Holm-adjusted

p

-value comparing the sPB of configurations (iii) and (i); bold values are significant at the

0.05

level.

Model name	Top- $k$
Model name	$k=5$	$k=10$	$k=15$	$k=20$
PS
DeepSeek R1 32B	21.9 (21.0, 22.7)	29.4 (28.5, 30.4)	34.1 (33.1, 35.2)	37.2 (36.2, 38.3)
DeepSeek R1 7B	34.0 (33.1, 34.9)	45.1 (44.2, 45.9)	52.7 (51.8, 53.6)	58.1 (57.3, 59.0)
DeepSeek R1 8B	29.8 (28.8, 30.7)	38.6 (37.7, 39.5)	43.2 (42.3, 44.2)	46.4 (45.5, 47.4)
LLaMA 3.1 8B	30.4 (29.5, 31.3)	41.9 (41.0, 42.8)	46.9 (46.0, 47.8)	49.6 (48.6, 50.5)

Appendix

Table 12: Permutation sensitivity (PS) and strong positional bias (sPB) across candidate-set sizes

k

at temperature

t=0

, with

95\%

confidence intervals obtained by percentile bootstrap at the query level. Values are percentages.

Model name	Top- $k$
	$k=5$		$k=10$		$k=15$		$k=20$
	Min.	Max.	Min.	Max.	Min.	Max.	Min.	Max.
DeepSeek R1 32B	86.8	87.4	80.6	80.8	76.7	77.1	74.0	74.7
DeepSeek R1 7B	78.4	78.8	67.3	68.3	60.6	61.1	55.0	55.7
DeepSeek R1 8B	80.8	81.6	72.9	73.5	68.8	69.5	65.9	66.9
LLaMA 3.1 8B	81.7	82.1	71.4	71.9	66.4	67.0	63.4	64.1

Appendix

Table 13: Minimum and maximum of the threshold

\tau

of sPB across the five folds, for candidate-set sizes

k

at temperature

t=0

. Values are percentages.

Model name	Temperature $t$
Model name	$t=0.0$	$t=0.2$	$t=0.4$	$t=0.6$
PS
DeepSeek R1 32B	21.9 (21.0, 22.7)	22.7 (21.8, 23.6)	22.9 (22.0, 23.8)	23.4 (22.5, 24.3)
DeepSeek R1 7B	34.0 (33.1, 34.9)	34.6 (33.8, 35.5)	35.3 (34.4, 36.2)	35.9 (35.0, 36.8)
DeepSeek R1 8B	29.8 (28.8, 30.7)	30.9 (30.0, 31.8)	31.5 (30.5, 32.3)	32.6 (31.7, 33.4)
LLaMA 3.1 8B	30.4 (29.5, 31.3)	31.4 (30.5, 32.3)	32.3 (31.4, 33.1)	33.0 (32.0, 33.8)

Appendix

Table 14: Permutation sensitivity (PS) and strong positional bias (sPB) across decoding temperatures

t

at top-

5

, with

95\%

confidence intervals obtained by percentile bootstrap at the query level. Values are percentages.

Model name	Temperature $t$
	$t=0.0$		$t=0.2$		$t=0.4$		$t=0.6$
	Min.	Max.	Min.	Max.	Min.	Max.	Min.	Max.
DeepSeek R1 32B	86.8	87.4	86.1	86.2	85.6	86.0	85.1	85.3
DeepSeek R1 7B	78.4	78.8	77.9	78.2	77.7	78.6	76.9	77.6
DeepSeek R1 8B	80.8	81.6	80.1	80.8	79.7	80.1	78.5	79.1
LLaMA 3.1 8B	81.7	82.1	80.2	80.9	79.4	79.7	79.0	79.6

Appendix

Table 15: Minimum and maximum of the threshold

\tau

of sPB across the five folds, for decoding temperatures

t

at top-

5

. Values are percentages.

	$k=5$		$k=10$		$k=15$		$k=20$
Model name	$\Delta$ sPB/SD (95% CI)	Rel.	$\Delta$ sPB/SD (95% CI)	Rel.	$\Delta$ sPB/SD (95% CI)	Rel.	$\Delta$ sPB/SD (95% CI)	Rel.
DeepSeek R1 32B	-0.08 (-0.25, 0.09)	-2.6%	-0.41 (-0.66, -0.16)	-5.8%	-0.58 (-0.89, -0.27)	-5.4%	-0.66 (-1.05, -0.26)	-4.8%
DeepSeek R1 7B	-0.26 (-0.58, 0.05)	-2.7%	-0.60 (-0.96, -0.23)	-3.0%	-1.09 (-1.49, -0.68)	-4.1%	-0.95 (-1.33, -0.56)	-2.9%
DeepSeek R1 8B	-0.30 (-0.54, -0.05)	-4.4%	-0.84 (-1.19, -0.49)	-5.7%	-0.59 (-0.98, -0.20)	-3.0%	-0.94 (-1.32, -0.55)	-4.5%
LLaMA 3.1 8B	-0.22 (-0.47, 0.03)	-3.3%	-0.81 (-1.17, -0.46)	-5.2%	-0.90 (-1.29, -0.50)	-4.3%	-0.60 (-1.02, -0.17)	-2.5%
LLaMA 4 Scout	-0.30 (-0.48, -0.12)	-9.3%	-0.85 (-1.22, -0.48)	-6.1%	-0.89 (-1.27, -0.51)	-3.7%	-1.08 (-1.44, -0.72)	-3.0%

Appendix

Table 16: Change of sPB, in percentage points, per standard deviation of retrieval score dispersion (95% CI), and relative to the sPB of each model (Rel.), across candidate-set sizes at

t=0

. Bold: significant at the

0.05

level after Holm correction.

	$t=0.0$		$t=0.2$		$t=0.4$		$t=0.6$
Model name	$\Delta$ sPB/SD (95% CI)	Rel.	$\Delta$ sPB/SD (95% CI)	Rel.	$\Delta$ sPB/SD (95% CI)	Rel.	$\Delta$ sPB/SD (95% CI)	Rel.
DeepSeek R1 32B	-0.08 (-0.25, 0.09)	-2.6%	-0.07 (-0.25, 0.11)	-2.2%	-0.19 (-0.37, -0.02)	-5.7%	-0.05 (-0.23, 0.13)	-1.5%
DeepSeek R1 7B	-0.26 (-0.58, 0.05)	-2.7%	-0.66 (-0.96, -0.36)	-6.6%	-0.74 (-1.04, -0.43)	-7.1%	-0.38 (-0.68, -0.07)	-3.6%
DeepSeek R1 8B	-0.30 (-0.54, -0.05)	-4.4%	-0.16 (-0.40, 0.09)	-2.2%	-0.42 (-0.72, -0.13)	-4.4%	-0.51 (-0.81, -0.20)	-5.1%
LLaMA 3.1 8B	-0.22 (-0.47, 0.03)	-3.3%	-0.33 (-0.58, -0.08)	-4.7%	-0.55 (-0.85, -0.26)	-5.9%	-0.17 (-0.46, 0.12)	-1.8%
LLaMA 4 Scout	-0.30 (-0.48, -0.12)	-9.3%	-0.17 (-0.34, 0.00)	-5.3%	-0.34 (-0.51, -0.17)	-10.4%	-0.29 (-0.46, -0.12)	-8.7%

Appendix

Table 17: As Table 16 , across decoding temperatures at

k=5

Model name	Metric	Top-5		Top-10		Top-15		Top-20
Model name	Metric	All	CC	All	CC	All	CC	All	CC
DeepSeek R1 32B	$L$	0.970%	0.563%	2.827%	2.291%	4.291%	4.114%	6.775%	6.618%
DeepSeek R1 32B	$p$ -value	7.456e-7	0.002	5.130e-17	7.082e-17	2.202e-47	2.072e-38	2.098e-48	2.859e-40
DeepSeek R1 7B	$L$	4.151%	5.279%	5.786%	7.430%	9.506%	9.432%	9.894%	9.969%
DeepSeek R1 7B	$p$ -value	6.197e-60	2.169e-64	9.362e-55	1.299e-45	5.966e-79	2.045e-47	4.449e-70	2.075e-33
DeepSeek R1 8B	$L$	0.788%	0.785%	1.959%	2.093%	4.303%	4.387%	7.238%	7.256%

Appendix

Table 18: Sensitivity of POB-UNBP to the discarded generations. For each model, the first row reports the lower bound

L

of the total variation distance to the uniform distribution and the second row the

p

-value of the POB-UNBP test, computed on all queries (All) and on the complete-case queries (CC), for which no generation was discarded.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 521 papers

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders
Jul 26, 2026Ge Zhang, Jingru Cheng, Huiyuan ChenLearning to Rank Adversarial Attacks
Stanford University · Independent Researcher
One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation
Apr 30, 2026Ethan Bito, Yongli Ren, Estrid HeLearning to Rank LLM Reranking
RMIT University, Melbourne, Australia
Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)
Mar 18, 2026Zichen Tang, Zirui Zhang, Qian Wang +3Large Language Model-Based Recommendation LLM Agent Reliability
The Hong Kong University of Science and Technology · National University of Singapore · The Hong Kong University of Science and Technology (Guangzhou)

Jul 26, 2026J/K move · Enter open · S save

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders

Ge Zhang, Jingru Cheng, Huiyuan Chen

Stanford University · Independent Researcher

Learning to Rank Adversarial Attacks Adversarial Attacks on LLMs LLM Reranking Recommender Systems Position Bias

Open paper AskPDF arXiv

Large language models (LLMs) used as listwise rerankers in recommendation systems suffer from position bias when serializing candidate sets into prompts. We show this order sensitivity creates an exploitable attack surface: an attacker can promote a label-0 target into the top-

k

solely by reordering candidates, without changing item content, labels, or model parameters. We introduce

\mathrm{promo}@k

to quantify this vulnerability, measuring the fraction of label-0 targets that can be elevated into top-

k

rankings via permutation. Evaluating across three domains (MovieLens, Amazon Books, and Amazon Fashion),

\mathrm{promo}@5

reaches up to 0.57 at an attack budget of

R

= 50 orderings. Furthermore, ordinary permutation stability predicts vulnerability without running the attack. While a bidirectional T5 encoder scorer reduces exposure, permutation-consistency regularization and architectural invariance effectively mitigate it. Pointwise scoring avoids the bias issue but degrades ranking quality. These results demonstrate that input candidate order in listwise LLM reranking is a security-relevant attack vector. Code and data are available at https://github.com/geoz-lab/position_bias_attack.

Abstract

Recommender Systems

Explore similar work

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)