Aligning Large Language Models with Searcher Preferences

Wei Wu; Peilun Zhou; Liyi Chen; Qimeng Wang; Chengqiang Lu; Yan Gao; Yi Wu; Yao Hu

cs.CLMar 11, 2026

Aligning Large Language Models with Searcher Preferences

Authors: Wei Wu, Peilun Zhou, Liyi Chen, Qimeng Wang, Chengqiang Lu, Yan Gao, Yi Wu, Yao Hu,

Organizations: School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei, China · Xiaohongshu Inc., Beijing, China · Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China

Abstract

The paradigm shift from item-centric ranking to answer-centric synthesis is redefining the role of search engines. While recent industrial progress has applied generative techniques to closed-set item ranking in e-commerce, research and deployment of open-ended generative search on large content platforms remain limited. This setting introduces challenges, including robustness to noisy retrieval, non-negotiable safety guarantees, and alignment with diverse user needs. In this work, we introduce SearchLLM, the first large language model (LLM) for open-ended generative search. We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints, including factual grounding, basic answer quality and format compliance, from behavior optimization objectives that promote robustness to noisy retrieval and alignment with user needs. Concretely, our reward model evaluates responses conditioned on the user query, session history, and retrieved evidence set, combining rule-based checks with human-calibrated LLM judges to produce an interpretable score vector over these dimensions. We introduce a Gated Aggregation Strategy to derive the training reward for optimizing SearchLLM with Group Relative Policy Optimization (GRPO). We deploy SearchLLM in the AI search entry of RedNote. Offline evaluations and online A/B tests show improved generation quality and user engagement, increasing Valid Consumption Rate by 1.03% and reducing Re-search Rate by 2.81%, while upholding strict safety and reliability standards.

Figures & tables

Figure 1. User interaction snapshots of open-ended generative search in RedNote. The bottom-right panel summarizes failure attribution from online user feedback.

Figure 2. Overview of the alignment framework for open-ended generative search. The pipeline incorporates a multi-dimensional reward system that explicitly decouples non-negotiable bottom-line constraints (Layer I) from behavioral optimization objectives (Layer II). A hybrid evaluation stack, consisting of deterministic rules and human-calibrated LLM judges, computes fine-grained scores across multiple dimensions. These signals are synthesized via a gated aggregation mechanism to stabilize the learning signal for Group Relative Policy Optimization (GRPO).

Layer	Req.	Subset	Reward Dimension	GenRM	Rubric	Ours
Behavioral	R1	Query	Query Satisfy	71.52	73.31	87.24
		Evidence	Reference Conflict	79.01	70.48	90.84
			Reference Irrelevant	88.92	83.63	94.96
Bottom-line	R2	Basic Quality	Answer Quality	64.02	65.69	82.43
			Self Consistency	59.68	64.22	78.20
			Multi-turn Repeat/Conflict	80.60	83.40	94.20

Table 1. Comparison on pointwise constraint of our reward system against multiple baselines. Metric reported is Accuracy.

Layer	Req.	Subset	Reward Dimension	GenRM	Rubric	Ours
Behavioral	R1	Query	Planning Quality	84.74	73.68	89.47
		Evidence	Reference Beneficiality	79.56	72.41	94.83
			Reference Diversity	72.60	96.44	97.86
			Reference Satisfaction	76.04	82.29	84.38
	R3	Richness	Claim Diversity	77.69	87.40	96.07
		Concise/ Usability	Answer Firstness	80.47	95.05	97.66

Table 2. Comparison on pairwise preference of our reward system against multiple baselines. Metric reported is AUC.

Method	AUC
GenRM	70.90
Rubric	72.13
Reward System (Ours)	86.48

Table 3. Comparison of holistic preference alignment.

	Robustness (R1)		Bottom-line (R2)			Alignment (R3)
Method	Query	Evidence	Basic	Hallu	Format	Rich	Usability
SFT	0.9167	0.5809	0.9675	0.9176	0.9697	0.9222	0.7939
RFT	0.9720	0.5880	0.9930	0.9260	0.9890	0.9600	0.8950
DPO	0.9540	0.6120	0.9885	0.9510	0.9745	0.9690	0.8610
GRPO-GenRM	0.9630	0.5980	0.9870	0.9340	0.9670	0.9840	0.8450
GRPO-Linear	0.9636	0.5861	0.9906	0.9714	0.9730	0.9738	0.8604

Table 4. Comparison of different optimization methods across all reward dimensions.

Figure 3. Comparison on generation quality of our policy against multiple baselines evaluated by human experts.

Figure 4. Training dynamics under different reward aggregation strategies. The curves illustrate the evolution of scores across distinct reward dimensions during training, comparing the Gated Aggregation strategy against the Linear baseline.

Figure 5. Results of the online A/B test on the RedNote platform conducted in 2026. The chart displays the relative changes in key user engagement metrics for our deployed model compared to the production baseline (SFT).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6. Case Study 1 highlighting improved temporal awareness for time sensitive queries

Panel of Figure 7 — Figure 7. Case Study 2 demonstrating the correction of factual hallucinations in domain specific queries

Panel of Figure 8 — Figure 8. Case Study 3 showcasing reduction of logical inconsistencies and redundancy in complex tasks

Figure 9. Screenshot of the human annotation interface. The workbench presents the query, user intent analysis, and reference notes. Annotators provide dual-granularity feedback: fine-grained scores on specific dimensions and holistic pairwise rankings.

Layer	Requirement	Subset	Reward Dimension	Implementation	Definition / Role
Layer I (Bottom-line)	R2	Hallucination	Highlight Hallucination	LLM-based	First sentence free of obvious hallucinations.
			Claim Hallucination	LLM-based	Non-hallucination at the claim level.
			LLM Knowledge	LLM-based	Checks knowledge correctness with external validation.
			No Supply Reject	LLM-based	Proper refusal behavior when no evidence is available.
		Basic Quality	Self Consistency	LLM-based	Logical self-consistency, no internal contradictions.
			Answer Quality	LLM-based	Avoids gibberish or extremely low-quality answers.

Appendix

Table 5. Detailed Definitions of Multi-Dimensional Reward Criteria.

Figure 10. Supplementary training dynamics showing the evolution of the remaining reward dimensions.

Personalized Language Model Alignment

Personalized language model alignment adapts model behavior to individual preferences while preserving safety, accuracy, and other shared constraints.

See all 61 papers

Controlling Output Rankings in LLM-Based Search via Generative Engine Optimization
Feb 3, 2026Haibo Jin, Huimin Zeng, Yifeng Luo +1Generative Engine Optimization Large Language Model-Based Recommendation
School of Information Sciences, University of Illinois at Urbana-Champaign, IL, USA
AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
Dec 18, 2025Tzu-Han Lin, Wei-Lin Chen, Chen-An Li +3Retrieval-Augmented Generation Knowledge Augmentation for Language Models
National Taiwan University · Department of Computer Science, University of Virginia
When Should LLMs Search? Counterfactual Supervision for Search Routing
Jul 7, 2026Minho KimKnowledge Augmentation for Language Models LLM Routing
1Sangmyung University · 2DMTLABS.

Feb 3, 2026J/K move · Enter open · S save

Controlling Output Rankings in LLM-Based Search via Generative Engine Optimization

Haibo Jin, Huimin Zeng, Yifeng Luo, Haohan Wang

School of Information Sciences, University of Illinois at Urbana-Champaign, IL, USA

Generative Engine Optimization Large Language Model-Based Recommendation

Open paper AskPDF arXiv

The way customers search for and choose products is changing with the rise of large language models (LLMs). LLM-based search, or generative engines, provides direct product recommendations to users, rather than traditional online search results that require users to explore options themselves. However, these recommendations are strongly influenced by the initial retrieval order of LLMs, which disadvantages small businesses and independent creators by limiting their visibility. In this work, we propose CORE, an optimization method that Controls Output Rankings in gEnerative Engines for LLM-based search. Since the LLM's interactions with the search engine are black-box, CORE targets the content returned by search engines as the primary means of influencing output rankings. Specifically, CORE optimizes retrieved content by appending strategically designed optimization content to steer the ranking of outputs. We introduce three types of optimization content: string-based, reasoning-based, and review-based, demonstrating their effectiveness in shaping output rankings. To evaluate CORE in realistic settings, we introduce ProductBench, a large-scale benchmark with 15 product categories and 200 products per category, where each product is associated with its top-10 recommendations collected from Amazon's search interface. Extensive experiments on four LLMs with search capabilities (GPT-4o, Gemini-2.5, Claude-4, and Grok-3) demonstrate that CORE achieves an average Promotion Success Rate of 91.4% @Top-5, 86.6% @Top-3, and 80.3% @Top-1, across 15 product categories, outperforming existing ranking manipulation methods while preserving the fluency of optimized content.

Abstract

Personalized Language Model Alignment

Explore similar work

Controlling Output Rankings in LLM-Based Search via Generative Engine Optimization

AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning

When Should LLMs Search? Counterfactual Supervision for Search Routing