Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?

Han Chen; Yingrui Li

cs.IRSep 25, 2026

Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?

Authors: Han Chen, Yingrui Li

Abstract

Behavioral tests measure how a language model reads evidence. We ask whether those measurements help choose a recommendation interface. We evaluate six small instruction-tuned checkpoints across four recommendation domains with chronological evaluation and 3,426 evaluation users. Each request ranks eight candidates. A baseline selector chooses among history-only prompting, prompting with collaborative evidence, and score fusion. It uses observable features and six stability prompts that vary wording and candidate order. An augmented selector adds features from six evidence-reading prompts that ask the model to compare support counts. An interface chosen once on development (validation) data for each domain and checkpoint scores 0.5524 NDCG@5, compared with 0.5447 for the baseline selector and 0.5428 for the augmented selector. Adding the diagnostic features changes NDCG@5 by -0.0019 (95% interval [-0.0046, 0.0004]). The interval includes zero, and its upper bound is below the analysis plan's 0.005 improvement target. Matching the selectors' hyperparameters also leaves the interval upper bound below that target. Evidence from retrieved similar users improves prompting by 0.0999 NDCG@5 over a control using randomly selected users matched for activity. The evidence-reading tests also reveal answer-position and tie-response biases. These results concern the tested selectors and candidate sets. They illustrate why diagnostic measurements should be evaluated by whether they improve recommendation choices beyond existing features and a fixed interface.

Figures & tables

Figure 1: Evidence-reading performance by checkpoint. Overall accuracy combines 254 direct and 258 pairwise evaluation cases. Pairwise balanced accuracy averages recall across left, right, and tie labels; its chance level is

1/3

. The two metrics have different denominators and scoring rules.

Domain	H	T	F	Dev-fixed	B	B+R	Oracle	BPR
MovieLens	.514	.636	.669	.669	.670	.669	.769	.698
Amazon	.509	.513	.523	.536	.532	.532	.619	.452
KuaiRec	.352	.424	.458	.458	.441	.436	.641	.546
Last.fm	.451	.518	.547	.547	.536	.535	.682	.538
Equal-domain	.456	.523	.549	.552	.545	.543	.678	.559

Table 1: NDCG@5 on the same candidate sets. Dev-fixed chooses one H/T/F interface per domain/checkpoint using development data; this reference was added post hoc. B and B+R are the baseline and augmented selectors. BPR averages five seeds. The last row weights domains equally after averaging checkpoints within users.

Figure 2: NDCG@5 changes and 95% intervals. H denotes domain-resampling hierarchical intervals; F denotes fixed-domain intervals; U denotes single-domain user-bootstrap intervals. Group exclusions use fixed-domain intervals, recomputing averages from the fitted policies’ outcomes without refitting. The matched-configuration comparison is post hoc. The dashed line marks the

+0.005

target.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Development	Evaluation	Zero support
MovieLens	498	1000	218
Amazon	500	1000	840
KuaiRec	500	1000	136
Last.fm	350	426	33

Appendix

Table 2: Cohort sizes and retrieved-evidence coverage. Zero support means zero retrieved counts for all eight candidates.

Metric	Group	Users	B	B+R	$\Delta$
NDCG@5	Full sample	426	0.536223	0.534711	$-0.001512$
NDCG@5	Retained	397	0.528338	0.527413	$-0.000925$
NDCG@5	Flagged	29	0.644166	0.634623	$-0.009543$
Hit@1	Full sample	426	0.288732	0.291080	$+0.002347$
Hit@1	Retained	397	0.280856	0.283375	$+0.002519$
Hit@1	Flagged	29	0.396552	0.396552	$+0.000000$

Appendix

Table 3: Last.fm timestamp groups. B denotes the baseline selector and B+R the augmented selector;

\Delta

is their paired difference.

Scope	Users retained	Scheme	$\Delta$	95% interval
Four domains	3426	H	$-0.001881$	$[-0.004580,0.000421]$
Four domains	3397	H	$-0.001735$	$[-0.004530,0.000636]$
Four domains	3426	F	$-0.001881$	$[-0.003971,0.000189]$
Four domains	3397	F	$-0.001735$	$[-0.003896,0.000382]$
Last.fm	426	U	$-0.001512$	$[-0.006856,0.003813]$
Last.fm	397	U	$-0.000925$	$[-0.006422,0.004534]$

Appendix

Table 4: NDCG@5 timestamp sensitivity. H, F, and U are defined above. The retained-user comparisons are post hoc; full-sample rows provide the reference.

Issue	Documented facts	Remaining uncertainty
Reused outcomes	All 1,000 MovieLens and 1,000 Amazon evaluation inputs match earlier objects. The analysis uses the same labels; earlier outcome summaries exist.	Whether and when earlier summaries were consulted during extension design. The September 18, 2026 plan has no precise finalization timestamp.
KuaiRec positives	An earlier feasibility record specifies watch ratio $>2.0$ . Both the preprocessing code and the scored cohorts use finite watch ratio $\geq 1.0$ .	The reason and timing of the change, and its relation to outcome inspection. No contemporaneous amendment was located.

Appendix

Table 5: Recorded analysis history and limits on confirmatory interpretation.

Checkpoint	Repository
Qwen2.5 1.5B	Qwen/Qwen2.5-1.5B-Instruct
Qwen2.5 3B	Qwen/Qwen2.5-3B-Instruct
Qwen3 4B	Qwen/Qwen3-4B-Instruct-2507
Phi-4 mini	microsoft/Phi-4-mini-instruct
Gemma-3 4B	unsloth/gemma-3-4b-it
Llama-3.2 3B	unsloth/Llama-3.2-3B-Instruct

Appendix

Table 6: Checkpoint repositories. Exact revisions are listed in the supplement.

Checkpoint	Overall	Direct	Pairwise
Qwen2.5 1.5B	26.17%	30.71%	21.71%
Qwen2.5 3B	41.21%	41.34%	41.09%
Qwen3 4B	58.59%	72.83%	44.57%
Phi-4 mini	54.69%	82.28%	27.52%
Gemma-3 4B	71.48%	85.43%	57.75%
Llama-3.2 3B	48.63%	72.83%	24.81%

Appendix

Table 7: Held-out test accuracy. Overall n=512, direct n=254, and pairwise n=258. The two decision formats use different primary cases.

Checkpoint	MovieLens	Amazon	KuaiRec	Last.fm
Qwen2.5 1.5B	0.8 Z	0.7 Z	1.0 M	0.7 Z
Qwen2.5 3B	0.7 Z	0.4 M	0.5 M	0.9 Z
Qwen3 4B	0.8 Z	0.6 M	1.0 M	0.7 Z
Phi-4 mini	0.9 Z	0.6 Z	1.0 M	0.8 Z
Gemma-3 4B	0.8 Z	0.6 Z	1.0 M	0.7 Z
Llama-3.2 3B	0.8 Z	0.6 Z	1.0 M	0.7 Z

Appendix

Table 8: Fusion coefficient and scaling selected from development utilities and applied unchanged to evaluation.

Inputs	Max. leaves	Min. leaf	L2	CV MSE
Observable only	4	80	10	0.129140
Observable + reading	8	80	1	0.129167
B	4	80	10	0.128892
B+R	8	80	10	0.128903

Appendix

Table 9: Development-selected cross-validation MSE and configurations. The mapping to archived feature keys is in the supplement README.

Domain	Mean	95% interval	90% interval
MovieLens 20M	$-0.001269$	$[-0.003592,0.000960]$	$[-0.003222,0.000622]$
Amazon Instruments	$-0.000463$	$[-0.003154,0.002247]$	$[-0.002742,0.001832]$
KuaiRec	$-0.004282$	$[-0.009660,0.001038]$	$[-0.008851,0.000237]$
Last.fm 1K	$-0.001512$	$[-0.006856,0.003813]$	$[-0.006067,0.002928]$
Pooled	$-0.001881$	$[-0.004580,0.000421]$	—

Appendix

Table 10: Change in selected NDCG@5 and interval containment. All four domain utility Holm-adjusted

p

-values equal 1.000. The pooled one-sided

p

-value is 0.962304.

Domain	Mean	95% interval	Holm $p$
MovieLens 20M	$-0.000142$	$[-0.001382,0.001067]$	1.000000
Amazon Instruments	$0.000364$	$[-0.000054,0.000787]$	0.299570
KuaiRec	$0.001095$	$[0.000201,0.001998]$	0.064794
Last.fm 1K	$0.000054$	$[-0.001290,0.001451]$	1.000000
Pooled	$0.000343$	$[-0.000362,0.001022]$	—

Appendix

Table 11: Reduction in two-target gain-forecast MSE. The last column reports the eight-test Holm correction.

Resampling	Mean	95% interval	90% interval
Primary hierarchical	$-0.001881$	$[-0.004580,0.000421]$	$[-0.004091,0.000085]$
Fixed domains; post hoc	$-0.001881$	$[-0.003971,0.000189]$	$[-0.003618,-0.000118]$

Appendix

Table 12: Pooled interval sensitivity. Hierarchical domain resampling is the primary; the fixed-domain interval is post hoc. Both operate on user-level checkpoint averages.

Domains	Users	Mean	95% interval
Last.fm	426	$-0.001512$	$[-0.006856,0.003813]$
KuaiRec + Last.fm	1426	$-0.002897$	$[-0.006713,0.000842]$
MovieLens + Amazon + Last.fm	2426	$-0.001081$	$[-0.003231,0.001055]$
MovieLens + Amazon	2000	$-0.000866$	$[-0.002651,0.000957]$

Appendix

Table 13: Domain restrictions. Last.fm uses the interval in Table 10 ; the other rows are post hoc fixed-domain user-bootstrap intervals.

Method	MovieLens	Amazon	KuaiRec	Last.fm
Popularity	0.3767 / 0.0750	0.3702 / 0.0990	0.3563 / 0.0490	0.3750 / 0.1197
Retrieved support	0.6094 / 0.3870	0.4208 / 0.1860	0.4579 / 0.2020	0.5208 / 0.2864
Random support	0.4025 / 0.1650	0.3841 / 0.1370	0.3945 / 0.1370	0.3234 / 0.1080
User-KNN	0.6111 / 0.3880	0.4203 / 0.1860	0.4585 / 0.1890	0.5263 / 0.2981
BPR (five seeds)	0.6978 / 0.4496	0.4521 / 0.1926	0.5461 / 0.2750	0.5384 / 0.2761
H	0.5138 / 0.2720	0.5087 / 0.2468	0.3515 / 0.1195	0.4513 / 0.1960

Appendix

Table 14: Conventional references, fixed interfaces, and primary selectors. Each cell is NDCG@5 / Hit@1 on identical candidates. Random support ranks candidates by random-neighbor counts; the LLM prompt control appears in Appendix F .

Contrast	Effect	95% interval
B+R minus B (primary)	$-0.001881$	$[-0.004580,0.000421]$
Reading + observable minus observable	$-0.000529$	$[-0.004761,0.003052]$
B minus observable	$0.002241$	$[-0.000042,0.004204]$
Reading + observable minus B (equal-prompt-count)	$-0.002770$	$[-0.006147,0.000303]$
B+R minus reading + observable	$0.000889$	$[-0.002034,0.004027]$
B+R minus B (Hit@1)	$0.000962$	$[-0.001643,0.003701]$

Appendix

Table 15: Feature-set comparisons. Positive effects favor the first-listed policy. All rows are secondary except the unchanged primary listed for context. The equal-prompt-count comparison uses six diagnostic prompts on each side. All intervals here use the hierarchical domain-resampling scheme.

Domain	B minus F	B+R minus F	B gap closed	B+R gap closed
MovieLens	$0.001113$	$-0.000155$	61.31%	60.81%
Amazon	$0.009447$	$0.008984$	21.22%	20.80%
KuaiRec	$-0.017562$	$-0.021844$	30.77%	29.29%
Last.fm	$-0.010416$	$-0.011927$	36.77%	36.12%
Equal-domain	$-0.004354$	$-0.006236$	39.94%	39.09%

Appendix

Table 16: Fixed-fusion reference and oracle gap closed relative to H. Always-F differs from Dev-fixed.

Domain	B: H	B: T	B: F	B+R: H	B+R: T	B+R: F
MovieLens	3.87	26.57	69.57	10.38	28.48	61.13
Amazon	15.72	53.53	30.75	33.65	48.70	17.65
KuaiRec	5.07	25.55	69.38	11.38	29.98	58.63
Last.fm	2.97	30.99	66.04	5.79	32.16	62.05
Equal-domain	6.91	34.16	58.94	15.30	34.83	49.87

Appendix

Table 17: Chosen-interface frequencies (percent). Users are averaged within domains and domains equally for the last row; checkpoints are repeated decisions.

Checkpoint	Accuracy	Mean confidence	ECE	Brier
Qwen2.5 1.5B	0.2617	0.6178	0.3583	0.9730
Qwen2.5 3B	0.4121	0.8190	0.4096	0.9647
Qwen3 4B	0.5859	0.9652	0.3854	0.8224
Phi-4 mini	0.5469	0.7647	0.2298	0.6789
Gemma-3 4B	0.7148	0.9847	0.2730	0.5998
Llama-3.2 3B	0.4863	0.7148	0.2456	0.7449

Appendix

Table 18: Held-out evidence-reading calibration, 512 primary cases per checkpoint. ECE uses ten confidence bins; different task formats are combined in the reported overall score.

Checkpoint	Tie	Other	Zero-mixed	Sparse	One-count	Dense
Qwen2.5 1.5B	0.4135	0.2230	0.1700	0.2136	0.2524	0.2549
Qwen2.5 3B	0.3462	0.4289	0.4300	0.5728	0.3204	0.3922
Qwen3 4B	0.4135	0.6299	0.6600	0.7282	0.5728	0.5588
Phi-4 mini	0.8750	0.4632	0.4600	0.5049	0.5049	0.3824
Gemma-3 4B	0.4615	0.7794	0.8700	0.9223	0.6796	0.6471
Llama-3.2 3B	0.7981	0.4069	0.3900	0.4951	0.3786	0.3627

Appendix

Table 19: Test accuracy by synthetic difficulty category. Tie n=104; all other difficulty categories combined n=408; zero-mixed n=100, sparse n=103, one-count n=103, dense-near-tie n=102. The difficulty category combines 50 direct and 54 pairwise cases; only the latter are explicit pairwise tie answers.

Category	MovieLens	Amazon	KuaiRec	Last.fm
All zero	218 / +0.0034	840 / +0.0005	136 / -0.0234	33 / -0.0146
Tied maximum (>0)	97 / -0.0004	15 / -0.0113	128 / -0.0057	76 / +0.0076
Sparse	126 / +0.0023	132 / -0.0047	111 / -0.0118	41 / -0.0016
One-count gap	132 / -0.0001	8 / -0.0128	169 / -0.0034	80 / -0.0085
Other mixed-zero	320 / -0.0052	5 / +0.0000	203 / -0.0025	61 / +0.0009
Other all-positive	107 / -0.0054	0 / —	253 / +0.0080	135 / -0.0004

Appendix

Table 20: Real-request difficulty breakdown: each cell is user count / mean primary NDCG@5 effect. The fixed precedence partitions requests; all-zero cases are split from positive tied maxima for readability.

Domain	Dev-fixed	B	B+R
MovieLens	0.668944 / 0.434333	0.670058 / 0.439833	0.668789 / 0.440333
Amazon	0.535911 / 0.285667	0.532019 / 0.285167	0.531556 / 0.286333
KuaiRec	0.458119 / 0.190500	0.440557 / 0.199667	0.436275 / 0.199500
Last.fm	0.546638 / 0.294210	0.536223 / 0.288732	0.534711 / 0.291080
Equal-domain	0.552403 / 0.301177	0.544714 / 0.303350	0.542833 / 0.304312

Appendix

Table 21: Evaluation performance of the development-selected fixed policy, compared with the two selectors. Each cell is NDCG@5 / Hit@1.

Contrast	Difference	95% interval
B minus fixed (NDCG@5)	$-0.007689$	$[-0.011056,-0.004270]$
B+R minus fixed (NDCG@5)	$-0.009570$	$[-0.013204,-0.005897]$
B minus fixed (Hit@1)	$0.002172$	$[-0.002023,0.006299]$
B+R minus fixed (Hit@1)	$0.003134$	$[-0.001389,0.007622]$

Appendix

Table 22: Paired fixed-policy comparisons, conditional on the four domains. The post hoc intervals use fixed-domain resampling.

Domain	T NDCG	Random NDCG	Difference	T Hit@1	Random Hit@1	Difference
MovieLens	0.635868	0.481366	0.154502	0.401167	0.229000	0.172167
Amazon	0.513007	0.486436	0.026572	0.275500	0.234500	0.041000
KuaiRec	0.424308	0.363141	0.061168	0.177333	0.126000	0.051333
Last.fm	0.518488	0.361146	0.157342	0.276213	0.119327	0.156886
Equal-domain	0.522918	0.423022	0.099896	0.282553	0.177207	0.105346

Appendix

Table 23: Retrieved (T) and activity-matched random-neighbor evidence prompts. The last row averages the four domain means equally.

Contrast	Difference	95% interval
Matched minus B (NDCG@5)	$-0.000430$	$[-0.001581,0.000706]$
Matched minus B+R (NDCG@5)	$0.001451$	$[-0.000594,0.003495]$
Matched minus B (Hit@1)	$0.000319$	$[-0.001018,0.001670]$
Matched minus B+R (Hit@1)	$-0.000643$	$[-0.003108,0.001775]$
B minus matched (MSE)	$-0.00000827$	$[-0.00008854,0.00007221]$
B+R minus matched (MSE)	$-0.00035110$	$[-0.00088771,0.00017566]$

Appendix

Table 24: Post hoc matched-configuration results with fixed-domain intervals. No new configuration search was performed.

	Pairwise			Direct
Checkpoint	Tie acc.	Non-tie acc.	False tie	Multiple	Unique
Qwen2.5 1.5B	35.19	18.14	69.12	48.00	26.47
Qwen2.5 3B	11.11	49.02	19.61	60.00	36.76
Qwen3 4B	1.85	55.88	0.98	84.00	70.10
Phi-4 mini	81.48	13.24	80.39	94.00	79.41
Gemma-3 4B	0.00	73.04	1.47	96.00	82.84

Appendix

Table 25: Actual label-based accuracies and false tie responses (percent). Pairwise denominators are 54 ties and 204 non-ties; direct denominators are 50 multiple maxima and 204 unique maxima.

Checkpoint	True left	True right	True tie
Qwen2.5 1.5B	(17, 15, 74)	(11, 20, 67)	(20, 15, 19)
Qwen2.5 3B	(77, 8, 21)	(56, 23, 19)	(38, 10, 6)
Qwen3 4B	(104, 0, 2)	(88, 10, 0)	(53, 0, 1)
Phi-4 mini	(24, 3, 79)	(10, 3, 85)	(7, 3, 44)
Gemma-3 4B	(104, 1, 1)	(51, 45, 2)	(49, 5, 0)
Llama-3.2 3B	(14, 8, 84)	(20, 10, 68)	(12, 2, 40)

Appendix

Table 26: Pairwise confusion counts. Each triple is predicted (left, right, tie); columns index the true answer. Each checkpoint has the same 106 left, 98 right, and 54 tie labels.

Checkpoint	Raw accuracy	Balanced accuracy	Left	Right	Tie
Qwen2.5 1.5B	21.71	23.88	18.60	19.38	62.02
Qwen2.5 3B	41.09	35.74	66.28	15.89	17.83
Qwen3 4B	44.57	36.72	94.96	3.88	1.16
Phi-4 mini	27.52	35.73	15.89	3.49	80.62
Gemma-3 4B	57.75	48.01	79.07	19.77	1.16
Llama-3.2 3B	24.81	32.50	17.83	7.75	74.42

Appendix

Table 27: Pairwise performance and predicted-answer frequencies (percent), computed from Table 26 .

Feature	Checkpoint	Conditional	Residual	Cells
Digits: valid mass	9.52	13.11	82.28	24
Digits: entropy	28.85	46.59	51.41	24
Digits: correct	4.86	6.92	89.66	22
Words: valid mass	10.43	14.03	78.60	24
Words: entropy	43.61	57.15	42.70	24
Words: correct	5.59	7.54	86.93	23

Appendix

Table 28: Reading-feature variance decomposition (percent). Checkpoint is the marginal share; conditional is the within-domain checkpoint share; residual is within-domain/checkpoint variance divided by total variance. Cells counts nonconstant domain/checkpoint cells out of 24. Denominators differ across the three percentage columns.

Family	Family effect	95% interval	Pooled without family
Qwen	$-0.000198$	$[-0.003039,0.002573]$	$-0.003565$
Phi	$-0.003007$	$[-0.007133,0.001168]$	$-0.001656$
Gemma	$-0.006795$	$[-0.011424,-0.002288]$	$-0.000899$
Llama	$-0.000893$	$[-0.004848,0.003069]$	$-0.002079$

Appendix

Table 29: Family-specific aggregate effects and pooled effects after leaving out the family. Qwen contains three checkpoints; other families contain one each.

Checkpoint	Test accuracy	Selection effect
Qwen2.5 1.5B	26.17%	$0.001303$
Qwen2.5 3B	41.21%	$0.000316$
Qwen3 4B	58.59%	$-0.002212$
Phi-4 mini	54.69%	$-0.003007$
Gemma-3 4B	71.48%	$-0.006795$
Llama-3.2 3B	48.63%	$-0.000893$

Appendix

Table 30: Checkpoint-level accuracy and selection effects. Each effect averages the same four domain means.

Item	Status
Primary, MSE, both rules	Both decision rules fail.
Hit@1, all four feature sets	Reported for all four feature sets, including the equal-prompt-count comparison; added interval specifications labeled secondary.
Fixed references, oracle gap closed	H/T/F, development-selected fixed policy, interface frequencies and oracle gap closed reported; ratio and best-fixed comparison are post hoc specifications.
Evidence specificity	Both random-support ranking and the distinct saved random-evidence prompt control reported.
Reading tests and calibration	Overall/difficulty category accuracy, ECE/Brier, actual tie-response cross-tab, forecast calibration and real-request difficulty counts reported.
Conventional baselines	Popularity, support, user-KNN and five-seed BPR on identical candidates.

Appendix

Table 31: Coverage of planned and post hoc analyses.

Boundary	Baseline	Baseline + reading
Shared main H/T prompts	8	8
Additional diagnostic prompts	6	12
Before shared random-control scoring	14	20
Including four shared random-control prompts	18	24

Appendix

Table 32: Required prompts per checkpoint–request; these are feature requirements rather than separate timed policy executions. The experiment scores the full prompt collection before policy selection.

Large Language Model-Based Recommendation

Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.

See all 213 papers

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLLM Evaluation Large Language Model-Based Recommendation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Reasoning with Evidence, Not Merely Rationales: Verifiable Preference Proofs for LLM-Based Recommendation
Oct 2, 2026Yu Hou, Nathaniel Kang, Pengkai Wang +1Large Language Model-Based Recommendation Explainable Recommendation
School of Mathematics and Computing (Computational Science and Engineering), Yonsei University Seoul, Republic of Korea · School of Computer Science and Engineering, Kyungpook National University Daegu, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University Suwon, Republic of Korea +1
Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation
May 27, 2026Annabella Sánchez-Guzmán, Lukas Eberhard, Denis Helic +1LLM Auditing LLM Prompting
Escuela Superior Polit´ecnica del Litoral · Graz University of Technology · Complexity Science Hub

Sep 1, 2026J/K move · Enter open · S save

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation

Zhongru Chen, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

LLM Evaluation Large Language Model-Based Recommendation LLM Reasoning

Open paper AskPDF arXiv

Large language models are increasingly used as interactive recommender assistants. Their evaluation should therefore go beyond plausible item recommendation and test whether they can recognize flawed recommendation requests. Existing recommender benchmarks mainly assess ranking, generation, or preference satisfaction, while existing error-detection benchmarks are usually not grounded in recommendation-specific user and candidate evidence. To address this gap, we introduce RPCBench, a benchmark for evaluating Recommender-Premise Critique: the ability to detect, diagnose, and properly handle faulty premises in natural-language recommendation requests. RPCBench contains evidence-grounded test instances from five recommendation domains and covers ten types of premise failures. Each instance provides a visible recommendation context and a corrupted user query. We further design a fine-grained evaluation framework that measures proactive detection, error localization, post-detection handling strategy, and evidence faithfulness. Through a systematic evaluation of 11 LLMs, we find that proactive detection is the main bottleneck in Recommender-Premise Critique, and models perform worst on underspecified-premise errors. We also observe that target-critical information density matters more than redundant evidence, and that longer reasoning does not monotonically improve critique quality: performance peaks at intermediate reasoning length, while overly long reasoning is accompanied by an overthinking penalty. The code is available at https://github.com/ZhongruChen/RPCBench.

Abstract

Large Language Model-Based Recommendation

Explore similar work

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation

Reasoning with Evidence, Not Merely Rationales: Verifiable Preference Proofs for LLM-Based Recommendation

Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation