ReflectRec: Cycle-Consistency Preference Verification for LLM-based Recommendation
Organizations: Sungkyunkwan University · NAVER Corp. · KIAT · KT Corp. ·
Abstract
CoT-style LLM recommenders generate a natural-language preference note as a chain-of-thought prefix to ranking, but the note is trusted without verification, and the ranker never consults behaviorally similar users. We propose ReflectRec, organized around a preference-to-history round-trip verifier whose recall score serves as an operational signal of preference-to-history recoverability, enabling conditional regeneration for low-recoverability preferences and selection among regeneration attempts. Around this verifier, we wire a semantic Stage-1 prompt, conditional regeneration, and Forward-Look retrieval that injects future watches of behaviorally similar users as a collaborative hint. On MovieLens-100K and MovieLens-1M, the full pipeline outperforms CoT4Rec-ICL by +16.25 / +4.46 pp Hit@10, with positive gains across all ValScore buckets and three LLM backbones (Qwen3-30B, Qwen2.5-72B, and Llama-3.3-70B). We also observe a similar verifier-guided regeneration pattern on Amazon Electronics.
Figures & tables
| Component | Setting |
|---|---|
| LLM (analyzer/ranker/verifier) | Qwen3-30B-A3B-Instruct |
| Retrieval embedder | BAAI/bge-large-en-v1.5 |
| Fragment size | 5 |
| Future window | 5 |
| Forward-look top- | 10 (single aspect) |
| Verifier candidates | 50 (true + random neg.) |
| Dataset | #Users | #Items | #Inter. | Sparsity | Deg.Cent. |
|---|---|---|---|---|---|
| ML-100K | 943 | 1,682 | 100,000 | 0.9370 | 106.04 |
| ML-1M | 6,040 | 3,416 | 999,611 | 0.9516 | 165.50 |
| Elec. | 1,500 | 55,588 | 45,000 | 0.9995 | 30.00 |
| Method | ML-100K | ML-1M | Electronics | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Hit@1 | Hit@5 | Hit@10 | Hit@1 | Hit@5 | Hit@10 | Hit@1 | Hit@5 | Hit@10 | |
| Trained baselines — leave-one-out | |||||||||
| GRU4Rec | 10.29 | 29.16 | 46.98 | 13.74 | 38.81 | 53.46 | 16.84 | 37.96 | 49.92 |
| SASRec | 20.89 | 56.10 | 72.75 | 37.81 | 71.85 | 82.14 | 22.28 | 48.01 | 59.21 |
| LLM-SRec | 9.86 | 31.28 | 48.04 | 17.38 | 45.89 | 62.30 | 22.18 | 49.70 | 62.79 |
| Trained baselines rerun — 100-candidate protocol | |||||||||
| Method | Hit@1 | Hit@5 | Hit@10 |
|---|---|---|---|
| ML-100K ( ) | |||
| CoT4Rec-ICL | 1.19 | 6.76 | 15.42 |
| Stage-1 (semantic) | 3.32 (+2.13) | 14.23 (+7.47) | 27.76 (+12.34) |
| w/ Regen | 3.44 (+0.12) | 14.23 ( ) | 28.35 (+0.59) |
| Full (w/ FL) | 3.08 (-0.36) | 16.61 (+2.38) | 31.67 (+3.32) |
| ML-1M ( ) | |||
| Condition | Hit@1 | Hit@5 | Hit@10 |
| Full pool, | |||
| Stage-1 only | 4.47 | 16.07 | 29.67 |
| w/ Regen | 4.47 (+0.00) | 16.20 (+0.13) | 29.87 (+0.20) |
| low bucket , | |||
| Stage-1 only | 3.81 | 13.33 | 22.86 |
| w/ Regen | 2.86 (-0.95) | 15.24 (+1.90) | 26.67 (+3.81) |
| Bucket | ML-100K | ML-1M | |
|---|---|---|---|
| low [0, 0.4) | 29 / 87 | ||
| mid [0.4, 0.5) | 84 / 145 | ||
| high [0.5, 0.7) | 301 / 525 | ||
| pass [0.7, 1.0] | 429 / 743 | ||
| Weighted (unified) | 843 / 1500 |
| Variant | ML-100K | ML-1M |
|---|---|---|
| Hit@10 | Hit@10 | |
| , 1 aspect | ||
| , 1 aspect | ||
| , 1 aspect (selected) | ||
| , 1 aspect | ||
| , 4 aspects (multi) |
| Backbone | Stage-1 | +Regen | Full |
|---|---|---|---|
| Qwen3-30B (ours) | 27.76 | 28.35 | 31.67 |
| Qwen2.5-72B | 28.00 | 28.59 | 39.03 |
| Llama-3.3-70B | 31.32 | 30.96 | 36.65 |
| vs Stage-1 | |||
| Qwen3-30B | – | ||
| Qwen2.5-72B | – |
| Hint source | ML-100K | ML-1M |
|---|---|---|
| None (Stage-1 + Regen) | ||
| Random-future control | ||
| w/ Forward-look retrieval | ||
| Net retrieval contribution |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Hit@10 | 95% CI | ||
|---|---|---|---|
| 0.3 | 32 | ||
| 0.4 | 116 | ||
| 0.5 | 345 | ||
| 0.6 | 684 | ||
| 0.7 | 1171 |
| Dataset | Hit@10 | 95% CI | ||
|---|---|---|---|---|
| ML-100K | 0.3 | 4 | ||
| 0.4 | 29 | |||
| 0.5 | 113 | |||
| 0.6 | 239 | |||
| 0.7 | 414 | |||
| ML-1M | 0.3 | 28 |
| Bucket | Dataset | Hit@10 | 95% CI | |
|---|---|---|---|---|
| Overall | ML-100K | 843 | ||
| ML-1M | 1500 | |||
| LOW | ML-100K | 29 | ||
| ML-1M | 87 | |||
| MID | ML-100K | 84 | ||
| ML-1M | 145 |
| Backbone | LOW | MID | HIGH | PASS |
|---|---|---|---|---|
| Qwen3-30B | ||||
| Qwen2.5-72B | ||||
| Llama-3.3-70B |
| Data | Mean A/B | B–A [95% CI] | LOW J. | ||
|---|---|---|---|---|---|
| ML-100K | 668 | .4260/.4331 | .0070 | .5944 | .4214 |
| ML-1M | 1,500 | .5185/.5187 | .0003 | .9399 | .7941 |
| Data | 30 items | 50 items | 100 items |
|---|---|---|---|
| ML-100K | .5390 | .4310 | .3110 |
| ML-1M | .6220 | .5343 | .4133 |
| Data | Method | H@1 | H@5 | H@10 | |
|---|---|---|---|---|---|
| ML-100K | User-kNN | 20 | 4.86 | 20.28 | 38.91 |
| User-kNN | 40 | 3.32 | 14.71 | 29.42 | |
| User-kNN | 80 | 1.54 | 10.79 | 22.89 | |
| Item-kNN | 5 | 1.30 | 4.98 | 9.25 | |
| Item-kNN | 10 | .83 | 4.51 | 8.42 | |
| ML-1M | User-kNN | 20 | 1.73 | 10.60 | 21.40 |
| ValScore bucket | ML-100K Hit@10 | ML-1M Hit@10 |
|---|---|---|
| LOW [0,.4) | 20.69 ( ) | 24.14 ( ) |
| MID [.4,.5) | 26.19 ( ) | 25.52 ( ) |
| HIGH [.5,.7) | 24.58 ( ) | 35.43 ( ) |
| PASS [.7,1] | 30.77 ( ) | 40.92 ( ) |
| Pearson | .044 | .115 |
| Decoder | ML-100K | ML-1M |
|---|---|---|
| Global popularity | 32.38 | 30.80 |
| Query-time-safe popularity | 37.13 | 30.13 |
| Forward-Look retrieval-only | 42.70 | 45.40 |
| Timestamp-safe Forward-Look retrieval-only | 40.69 | 42.87 |
Large Language Model-Based Recommendation
Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.
See all 238 papers