cs.IRNov 13, 2026

Label-Free Detection of Structural Bias in LLM-Based Recommender Systems via Randomized Paraphrasing: The Case of Verbosity

Authors: Carlos Peláez-González, David Herrera-Poyatos, Francisco Herrera, Andrés Herrera-Poyatos

Organizations: Department of Computer Science and Artificial Intelligence (DECSAI), University of Granada, 18014 Granada, Spain. · Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI), 18016 Granada, Spain. · Department of Algebra, Faculty of Sciences, University of Granada, 18071 Granada, Spain.

Abstract

Large language models (LLMs) are increasingly deployed as recommenders in e-commerce and conversational shopping systems, where they select a product from a set of retrieved candidates described in natural language. Their decisions should depend on relevance to the user query, but may also be influenced by surface-level properties of the descriptions, such as their length. Detecting such structural biases is difficult because relevance labels are rarely available. We propose a label-free framework to audit LLM-based recommender systems for structural bias. Candidate descriptions are rewritten by randomly assigned paraphrasing models and presented in random order. We prove that, if the recommender's choice is invariant to these rewritings, every paraphraser is selected with equal probability, yielding a statistical test that requires no relevance labels. A four-step statistical pipeline tests this property and quantifies the contribution of each description feature while controlling for query-product relevance. We apply the framework to verbosity bias on 20{,}000 product queries and nine recommender LLMs from five model families. After controlling for the measured relevance signals, description length has a positive effect on selection in all nine models, increasing the odds of selection by up to 38% per standard deviation of length. Relevance remains the primary driver of recommendations, while verbosity acts as a consistent secondary signal whose strength depends on the model family. Since the framework needs only black-box access and no annotations, it can be run as a routine audit of deployed shopping assistants, and we provide an auditing protocol with cost and sample-size guidance.

Figures & tables

Explore similar work

CardsList
  1. LBR: Towards Mitigating Length Bias in Large Language Models for Recommendation

    Jul 5, 2026Hongchen Li, Bohao Wang, Jingbang Chen +5Large Language Model-Based RecommendationAttention Mechanisms

  2. Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence

    Oct 27, 2026Carlos Peláez-González, Francisco Herrera, Andrés Herrera-PoyatosRecommender SystemsLarge Language Model-Based Recommendation

  3. Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)

    Mar 18, 2026Zichen Tang, Zirui Zhang, Qian Wang +3Large Language Model-Based RecommendationLLM Agent Reliability