Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis
Organizations: Nikkei Inc. Tokyo, Japan
Abstract
Text-based generative recommenders can go beyond ranking existing items by directly generating item titles or other textual representations. In the news domain, this raises a societal concern that models may repeatedly surface already popular articles and thereby narrow the diversity of information to which users are exposed. We investigate training-data memorization as one mechanism behind this concentration. Using approximately 20 million browsing sessions from a Japanese news service, we continually trained an instruction-tuned Llama 3 8B model to generate the next article title in a session. The generated-title distribution was more concentrated and less diverse than the distribution of titles observed in the browsing logs. Frequently repeated target titles exhibited greater verbatim and approximate memorization, and more strongly memorized titles were generated in response to a wider range of prompts. These findings support a pathway in which popularity in behavioral logs becomes repeated training text, is preferentially memorized, and is subsequently reproduced by a generative recommender. An exploratory target-level deduplication condition reduced both memorization and output concentration, while substantially reducing the training-set size and recommendation accuracy. Our results suggest that popularity bias in generative recommendation may arise not only from user preferences or ranking objectives, but also from the model's tendency to reproduce frequently encountered strings, with important implications for information diversity on news platforms.
Figures & tables
| Measure | Prompt | Reference | Generation |
|---|---|---|---|
| Concentration | 0.8180 | 0.9003 | 0.9072 |
| Entropy | 10.710 | 9.147 | 7.169 |
| GS score | 0.1736 | 0.1911 | 0.3364 |
| Distinct titles | 193,815 | 101,026 | 20,689 |
| Prompt | Ref. freq. | Reference | Generated title | Gen. freq. | Verbatim | Approx. |
|---|---|---|---|---|---|---|
| 90切りの現実考察とそれを可能にする練習法(中) | 10 | 90切りの現実考察とそれを可能にする練習法(上) | 90切りの現実考察とそれを可 能にする練習法(上) | 22 | 24 | 1.000 |
| 90切りの現実考察とそれを可能にする練習法(上) | 11 | 90切りの現実考察とそれを可能にする練習法(下) | 90切りの現実考察とそれを可 能にする練習法(下) | 11 | 24 | 1.000 |
| 90切りの現実考察とそれを可能にする練習法(下) | 12 | 90切りの現実考察とそれを可能にする練習法(中) | 90切りの現実考察とそれを可 能にする練習法 (上) | 22 | 22 | 0.958 |
| 中国、制限なし春節スタート 旅客倍増で感染拡大警戒 | 141 | その決断、本当に自分の意思? 選択より前に脳が反応 | その決断、本当に自分の意思? 選択より前に 「アジッド」 | 25840 | 21 | 0.777 |
| Twitter社員、買収前の5分の1に 1300人に減少と米報道 | 3103 | その決断、本当に自分の意思? 選択より前に脳が反応 | その決断、本当に自分の意思? 選択より前に 「アジッド」 | 25840 | 21 | 0.777 |
| Condition | Conc. | Entropy | GS | Types | Mean verbatim memorization | Mean approximate memorization | : verbatim memorization | : approximate memorization | Exact-match correct |
|---|---|---|---|---|---|---|---|---|---|
| Original | 0.9072 | 7.169 | 0.3364 | 20,689 | 0.1356 | 0.0360 | 0.1965 | 0.3084 | 2,461 |
| Target-deduplicated | 0.8901 | 7.222 | 0.4658 | 39,479 | 0.0164 | 0.0264 | 0.0692 | 0.0395 | 81 |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 578 papers