Label-Free Detection of Structural Bias in LLM-Based Recommender Systems via Randomized Paraphrasing: The Case of Verbosity
Authors: Carlos Peláez-González, David Herrera-Poyatos, Francisco Herrera, Andrés Herrera-Poyatos
Organizations: Department of Computer Science and Artificial Intelligence (DECSAI), University of Granada, 18014 Granada, Spain. · Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI), 18016 Granada, Spain. · Department of Algebra, Faculty of Sciences, University of Granada, 18071 Granada, Spain.
Large language models (LLMs) are increasingly deployed as recommenders in e-commerce and conversational shopping systems, where they select a product from a set of retrieved candidates described in natural language. Their decisions should depend on relevance to the user query, but may also be influenced by surface-level properties of the descriptions, such as their length. Detecting such structural biases is difficult because relevance labels are rarely available. We propose a label-free framework to audit LLM-based recommender systems for structural bias. Candidate descriptions are rewritten by randomly assigned paraphrasing models and presented in random order. We prove that, if the recommender's choice is invariant to these rewritings, every paraphraser is selected with equal probability, yielding a statistical test that requires no relevance labels. A four-step statistical pipeline tests this property and quantifies the contribution of each description feature while controlling for query-product relevance. We apply the framework to verbosity bias on 20{,}000 product queries and nine recommender LLMs from five model families. After controlling for the measured relevance signals, description length has a positive effect on selection in all nine models, increasing the odds of selection by up to 38% per standard deviation of length. Relevance remains the primary driver of recommendations, while verbosity acts as a consistent secondary signal whose strength depends on the model family. Since the framework needs only black-box access and no annotations, it can be run as a routine audit of deployed shopping assistants, and we provide an auditing protocol with cost and sample-size guidance.
Figures & tables
Figure 1: Evaluation framework for structural bias detection. For a sampled query q , the retriever Rak returns the top- k products x1,…,xk from the catalog X . Each product xi is rewritten by its assigned paraphraser, x~i=τφ(i)(xi) , and the rewritten products are presented to the recommender in the order x~π(1),…,x~π(k) , where φ and π are independent, uniformly sampled permutations. The recommender returns a prompt position i , from which the selected paraphraser is recovered as m=φ(π(i)) . In the example shown, k=5 , the paraphrasers are τ1= DeepSeek-R1 7B (Qwen), τ2= GPT-OSS 20B, τ3= Gemma-3 12B, τ4= Falcon-3 7B and τ5= the identity paraphraser, and the sampled permutations are given by (φ(1),…,φ(5))=(1,5,3,2,4) and (π(1),…,π(5))=(3,1,5,4,2) . For instance, if the recommender selects prompt position i=2 , the recommended product is xπ(2)=x1 , which was rewritten by τφ(1)=τ1 , so m=1 .
Figure 2: Products retrieved using the BAAI/bge-small-en-v1.5 ranker, where x1..k=Rak(q,X) and X is a subset of the Amazon ML Challenge 2023 dataset. We only display the product descriptions although the products are composed by both title and description.
Figure 3: Product descriptions retrieved as described in Figure 2 and paraphrased by τ1=DeepSeek-R1 7B (Qwen) , τ2=GPT-OSS 20B , τ3=Gemma-3 12B , τ4=Falcon-3 7B , and τ5=Identity paraphraser . We shuffle the paraphrasers using the sampled permutation φ given by (φ(1),…,φ(5))=(1,5,3,2,4) , so that product xi is rewritten by paraphraser τφ(i) .
Figure 4: Feature distributions across paraphraser models for the sampled Amazon ML Challenge 2023 dataset (see subsection 4.1 ). Each column corresponds to a surface-level or semantic feature; each row to a paraphraser. ϕLEN measures description length in tokens. ϕADJ (adjective ratio) is the fraction of words that are adjectives; it is shown as an example of an additional surface-level extractor but is not used in the analysis. ϕCES measures query–product relevance. ϕSIM measures paraphrased–original product cosine similarity. The identity paraphraser ( Original ) yields a degenerate ϕSIM distribution (similarity always 1), shown with a truncated bar. Red dashed lines represent the median.
Figure 5
Recommender LLM
Paraphrasers
p -value
DeepSeek-R1
Falcon-3
GPT-OSS
Gemma-3
Original
DeepSeek-R1 7B (Qwen)
-1.13%
-0.53%
+0.11%
+0.49%
+1.06%
0.000029
DeepSeek-R1 8B (Llama)
-0.81%
-0.62%
+1.18%
+0.46%
-0.21%
0.000025
Falcon-3 10B
+0.19%
-0.20%
+0.43%
+0.85%
-1.28%
0.000050
Falcon-3 7B
-0.20%
-0.01%
+0.03%
+1.28%
-1.09%
0.000018
GPT-OSS 120B
-0.12%
-0.07%
+0.14%
+0.69%
-0.64%
0.059720
Table 1: The differences p(m)−1/k in our experiments, where the recommender LLM chooses products paraphrased by k specific models ( k=5 ), as specified in Equation 2 . A Chi-squared test is performed on the probability of choosing a paraphraser over the uniform distribution, whose results are the provided p -values. p -value lower than 0.05 (bold) indicates a bias.
Recommender LLM
CES
RAN
SIM
LEN
Solo
Unique
Solo
Unique
Solo
Unique
Solo
Unique
DeepSeek-R1 7B (Qwen)
5.99%
2.24%
3.34%
0.06%
0.14%
0.03%
3.15%
2.25%
DeepSeek-R1 8B (Llama)
7.26%
3.10%
3.69%
0.04%
0.09%
0.03%
2.49%
1.69%
Falcon-3 10B
7.71%
4.07%
3.27%
0.00%
0.02%
0.01%
0.94%
0.45%
Falcon-3 7B
7.68%
3.61%
3.64%
0.01%
0.02%
0.04%
1.24%
0.70%
GPT-OSS 120B
8.32%
3.66%
4.41%
0.08%
0.02%
0.01%
0.70%
0.28%
Table 2: Product feature contributions to different recommender LLMs decision rules ( R2 ). CES: Cross-Encoder Score; RAN: bi-encoder query–product cosine similarity. SIM: similarity to original description. LEN: description token length. Solo: marginal R2 of each feature alone; Unique: R2 after controlling for all other features.
Recommender LLM
βlensolo
Coefficients
Atten.
βCESunique
βRANunique
βSIMunique
βLENunique
DeepSeek-R1 7B (Qwen)
+0.3505
+0.5461
+0.0975
-0.0407
+0.3220
8.15%
DeepSeek-R1 8B (Llama)
+0.3119
+0.6471
+0.0755
-0.0402
+0.2811
9.87%
Falcon-3 10B
+0.1930
+0.7373
-0.0102
-0.0231
+0.1469
23.90%
Falcon-3 7B
+0.2221
+0.6939
+0.0436
-0.0427
+0.1826
17.79%
GPT-OSS 120B
+0.1672
+0.7000
+0.1065
-0.0245
+0.1167
30.17%
Table 3: β coefficients from a CLogit predicting paraphraser selection. A positive (negative) value indicates that higher values of the variable increase (decrease) the probability of the product being selected by the recommender LLM. βlensolo : length coefficient estimated without controls. Atten.: percentage reduction in βlen when all controls are included.
Figure 5: Histograms of the feature differences between each original product description and its paraphrase, one panel per LLM paraphraser τc . For each query q and each candidate product x rewritten by τc , the original description x is compared with its paraphrase τc(x) through ΔLEN=log(ϕLEN(x)/ϕLEN(τc(x))) (top) and ΔCES=ϕCES(q,x)−ϕCES(q,τc(x)) (bottom), using the same log-scale differences as Step 4. Positive values mean that the original description is longer or more relevant than its paraphrase. Red dashed lines mark the median, orange dotted lines the 2.5th and 97.5th percentiles.
Recommender LLM
Both signals clear
One signal clear
(++)
(+−)
(−+)
(−−)
(+=)
(=+)
(=−)
(−=)
Selection gap
DeepSeek-R1 7B (Qwen)
+20.2%
-0.3%
+2.4%
-16.8%
+4.9%
+10.9%
-12.1%
-1.9%
DeepSeek-R1 8B (Llama)
+18.1%
-4.0%
+4.0%
-18.0%
+2.7%
+10.6%
-12.8%
-1.3%
Falcon-3 10B
+13.8%
-8.4%
+6.4%
-15.8%
+2.8%
+12.3%
-13.4%
-0.8%
Falcon-3 7B
+15.2%
-7.5%
+6.0%
-17.1%
+2.2%
+12.8%
-12.5%
-1.8%
GPT-OSS 120B
+14.8%
-9.9%
+10.1%
-14.3%
+0.9%
+12.3%
-13.0%
+1.9%
Table 4: Average selection probability gap ( Equation 8 ) between the focal paraphraser (identity) and the four comparison paraphrasers, stratified by joint LEN–CES conditions. Each condition is a pair in {+,−,=}2 : the first coordinate refers to description length ( ϕLEN ) and the second to the cross-encoder score ( ϕCES ). For each query q and comparison paraphraser c , a + ( − ) indicates that the difference Δ between the original description xq,f and the paraphrased description xq,c is larger than log(1.1) (smaller than −log(1.1) ), where Δ is the log-ratio of lengths for ϕLEN and the difference of cross-encoder logits for ϕCES ; an = indicates that ∣Δ∣≤log(1.1) . Green cells indicate that the original description is preferred by the recommender, and red cells that it is avoided. The minimum samples block reports, for each recommender model and condition, the smallest number of queries satisfying the condition across the four comparison paraphrasers.
State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, Zhejiang, China +3
Department of Computer Science and Artificial Intelligence, Andalusian Institute of Data Science and Computational Intelligence (DaSCI), University of Granada, Spain. · Department of Algebra, University of Granada, Spain.