Merge and Branch: Efficient Tree-Structured Supervised Fine-Tuning of LLMs for Recommendation

Yunqin Zhu; Yao Xie

cs.IROct 27, 2026

Merge and Branch: Efficient Tree-Structured Supervised Fine-Tuning of LLMs for Recommendation

Authors: Yunqin Zhu, Yao Xie

Organizations: Georgia Institute of Technology, Atlanta, GA, USA

Abstract

LLM-based generative recommendation reads long, text-rich interaction histories but generates only short semantic identifiers (SIDs). This asymmetry makes supervised fine-tuning inefficient: encoding the history is expensive, yet each history usually provides supervision for only one target. We present Merge and Branch (M&B) to reduce input cost and learn more from each encoded history. Merge compresses each historical item's SID and text into a small number of learned tokens. Branch uses the compressed history to predict multiple target SIDs. To share history computation while keeping these targets independent, M&B organizes each example as a shallow, wide tree. We develop tree FlashAttention to execute this layout efficiently, processing the shared history once and propagating gradients from every branch for end-to-end training. A tree-structured key/value cache extends history sharing to constrained decoding. Experiments on public benchmarks show that M&B outperforms strong recommendation baselines while substantially improving training and decoding efficiency.

Figures & tables

Figure 1. Generative recommendation paradigms. M&B pairs compact SID–text inputs with independent SID targets. Three rows compare SID history to one SID target, SID and text history to one SID target, and merged SID and text tokens to parallel SID targets.

Method	Fwd.	Bwd.	Cache	SFT	Beam
SGLang ( Zheng et al., 2024b )			$\boldsymbol{\checkmark}$
DeFT ( Yao et al., 2025 )	$\boldsymbol{\checkmark}$		$\boldsymbol{\checkmark}$
Tree Training ( Wang et al., 2026 )	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$		$\boldsymbol{\checkmark}$
AtSpeed ( Lin et al., 2025b )					$\boldsymbol{\checkmark}$
UniSGR ( Sun et al., 2026 )			$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$
M&B	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$

Table 1. Support for tree-structured training and decoding. Fwd./Bwd.: FA-style forward/backward kernels, excluding dense masks and path traversal approaches.

Figure 2. Overview of M&B. A lightweight cross-attention merges each interaction into

N_{q}

soft tokens, which join the instruction and optional user profile to form the LLM prompt. Tree FlashAttention shares this prefix across independent primary, future, and synthetic SID branches during training, while KV caching reuses the prefix during beam search. A text-rich interaction history is compressed item by item into ordered soft tokens. An inset shows SID, action, and time information conditioning queries that attend to SID and text keys and values. The soft tokens join the task instruction and optional user profile in a shared LLM prefix. Primary, future, and synthetic SID branches read this prefix independently. The tree attention matrix shows causal attention within the prefix and within each branch, with no attention between sibling branches. Online softmax supports attention computation, and KV caching shares the prefix during decoding.

Dataset	# Users	# Items	# Inters.	Len.	Density
Beauty	22,363	12,101	198,502	8.9	0.0734%
Sports	35,598	18,357	296,337	8.3	0.0453%
Toys	19,412	11,924	167,597	8.6	0.0724%

Table 2. Dataset statistics. Len.: mean interactions per user; Density: observed fraction of user–item pairs.

Family	Method	Beauty				Sports				Toys
Family	Method	R@5	N@5	R@10	N@10	R@5	N@5	R@10	N@10	R@5	N@5	R@10	N@10
ID-based	SASRec	0.0537	0.0376	0.0788	0.0456	0.0293	0.0198	0.0432	0.0243	0.0661	0.0479	0.0860	0.0543
	SASRec-CE	0.0680	0.0496	0.0932	0.0577	0.0375	0.0269	0.0537	0.0322	0.0745	0.0554	0.0988	0.0633
	BERT4Rec	0.0372	0.0260	0.0554	0.0319	0.0161	0.0106	0.0251	0.0135	0.0338	0.0242	0.0475	0.0286
SID-based	TIGER	0.0454	0.0321	0.0648	0.0384	0.0264	0.0181	0.0400	0.0225	0.0521	0.0371	0.0712	0.0432
	LETTER	0.0431	0.0286	0.0672	0.0364	0.0275	0.0180	0.0428	0.0229	0.0436	0.0294	0.0679	0.0372

Table 3. Next-item recommendation on Amazon. R/N: Recall/NDCG. Underline: best baseline; bold: best overall. Improv. : relative gain over the strongest baseline.

Figure 3. Component ablations on Beauty (

N_{q}=4

). S+T: SID+Text; M: token merging; +FB/+SB: future/synthetic supervision; Full: both. Error bars: SD over five seeds. Bar chart of NDCG at rank ten for SID, text, and combined inputs, followed by token merging and future or synthetic supervision. Error bars show variation across runs.

Panel of Figure 4 — Figure 4. Token-budget ablation on Beauty. Reference lines: uncompressed SID+Text (smaller micro-batch, same effective batch). Error bars: SD over five seeds. Two panels compare recommendation accuracy and training cost as the number of merged tokens per item increases. The accuracy panel shows NDCG and Recall at rank ten; the efficiency panel shows optimizer-step time and peak memory. Dashed horizontal lines mark the uncompressed reference.

Figure 5. Target layout and loss weighting on Beauty (

N_{q}=1

). Each marker represents one of five seeds; Uniform/Weighted denote uniform/group weighting. Four panels compare Recall and NDCG for causal and tree target layouts under uniform and group-weighted losses. Box plots summarize the runs, and individual markers show each run.

Figure 6. Attention forward–backward time and memory on H200 (

B=128

). Means over five repeats; memory: peak allocated; OOM: out of memory. Causal FA uses a causal mask rather than a tree mask. Two line charts compare attention forward-backward time and peak memory as context length increases. Standard attention, Causal FA, and Tree FA use distinct line and hollow-marker styles. The longest standard-attention case is marked out of memory.

$T$	Causal	Tree	Improv.	Causal	Tree	Improv.
	Latency (ms/request) $\downarrow$			Memory (GiB) $\downarrow$
64	86.04 $\,\pm\,0.48$	77.12 $\,\pm\,1.55$	10.4%	4.02 $\,\pm\,0.00$	3.31 $\,\pm\,0.00$	17.6%
128	86.41 $\,\pm\,0.71$	76.33 $\,\pm\,0.62$	11.7%	4.70 $\,\pm\,0.00$	3.32 $\,\pm\,0.00$	29.4%
256	86.33 $\,\pm\,0.44$	77.06 $\,\pm\,0.41$	10.7%	6.07 $\,\pm\,0.00$	3.34 $\,\pm\,0.00$	45.0%
512	87.31 $\,\pm\,0.99$	78.04 $\,\pm\,1.42$	10.6%	8.86 $\,\pm\,0.00$	3.39 $\,\pm\,0.00$	61.8%
1024	91.13 $\,\pm\,0.39$	76.44 $\,\pm\,0.33$	16.1%	14.43 $\,\pm\,0.00$	3.46 $\,\pm\,0.00$	76.0%

Table 4. SID beam search on H200 (

B=1

, 100 beams, context length

T

). Causal: causal FA with duplicated per-beam KV caches; Tree: tree FA with a shared-prefix KV cache. Mean

\pm

SD over five repeats; memory: total peak allocated. Improv. : reduction relative to Causal.

Figure 7. Training curves on Beauty with different backbones. Vertical dashed lines mark the boundaries of epochs 1–4; markers denote validation-selected checkpoints. Training loss and test NDCG are plotted against optimizer steps for Qwen3, LLaMA, and T5 backbones and pretraining or instruction ablations. An enlarged early-stage panel compares the Qwen3-1.7B variants. Vertical dashed lines mark epoch boundaries, and hollow markers identify validation-selected checkpoints.

Panel of Figure 8 — Figure 8. Embedding distributions on Beauty. SID and merged embeddings are projected into the t-SNE space fitted to text embeddings. Colors indicate the data split where items first appear; Unseen denotes tokens absent from the data. Three scatter plots show SID, text, and merged representations in a common t-SNE coordinate system. Colors and markers distinguish training, validation, test, and unseen tokens; merged item representations from different data splits overlap.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Context length	Time (ms)			Memory (GiB)
	Standard attention	Tree FA	Causal FA	Standard attention	Tree FA	Causal FA
64	1.564 $\pm$ 0.042	0.590 $\pm$ 0.001	0.545 $\pm$ 0.003	0.475 $\pm$ 0.000	0.502 $\pm$ 0.000	0.502 $\pm$ 0.000
128	2.112 $\pm$ 0.001	1.160 $\pm$ 0.002	0.980 $\pm$ 0.002	1.082 $\pm$ 0.000	0.878 $\pm$ 0.000	0.878 $\pm$ 0.000
256	5.925 $\pm$ 0.001	2.237 $\pm$ 0.002	1.872 $\pm$ 0.001	3.051 $\pm$ 0.000	1.505 $\pm$ 0.000	1.505 $\pm$ 0.000
512	19.277 $\pm$ 0.000	4.712 $\pm$ 0.003	4.080 $\pm$ 0.002	9.989 $\pm$ 0.000	2.759 $\pm$ 0.000	2.759 $\pm$ 0.000
1,024	71.106 $\pm$ 0.015	11.404 $\pm$ 0.132	10.513 $\pm$ 0.182	35.865 $\pm$ 0.000	5.268 $\pm$ 0.000	5.268 $\pm$ 0.000

Appendix

Table 5. Attention forward–backward cost on H200 (

B=128

). Mean

\pm

SD over five repeats; memory: peak allocated; OOM: out of memory. Causal FA uses a causal mask.

Input	M	FB	SB	Recall@5	NDCG@5	Recall@10	NDCG@10	Time (ms)	Memory (GB)
SID				0.0555 $\,\pm\,0.0009$	0.0390 $\,\pm\,0.0006$	0.0786 $\,\pm\,0.0011$	0.0464 $\,\pm\,0.0006$	279.92 $\,\pm\,10.07$	42.96 $\,\pm\,0.00$
Text				0.0584 $\,\pm\,0.0015$	0.0413 $\,\pm\,0.0008$	0.0837 $\,\pm\,0.0015$	0.0494 $\,\pm\,0.0009$	3611.06 $\,\pm\,71.56$ ∗	99.19 $\,\pm\,0.00$ ∗
SID + Text				0.0594 $\,\pm\,0.0006$	0.0421 $\,\pm\,0.0005$	0.0842 $\,\pm\,0.0014$	0.0500 $\,\pm\,0.0006$	3674.85 $\,\pm\,72.76$ ∗	99.19 $\,\pm\,0.00$ ∗
SID + Text	$\checkmark$			0.0546 $\,\pm\,0.0011$	0.0375 $\,\pm\,0.0007$	0.0808 $\,\pm\,0.0005$	0.0459 $\,\pm\,0.0005$	289.57 $\,\pm\,1.02$	44.16 $\,\pm\,0.04$
SID + Text	$\checkmark$	$\checkmark$		0.0554 $\,\pm\,0.0010$	0.0381 $\,\pm\,0.0008$	0.0819 $\,\pm\,0.0014$	0.0466 $\,\pm\,0.0007$	295.12 $\,\pm\,1.08$	45.19 $\,\pm\,0.04$
SID + Text	$\checkmark$		$\checkmark$	0.0719 $\,\pm\,0.0008$	0.0505 $\,\pm\,0.0006$	0.0982 $\,\pm\,0.0005$	0.0589 $\,\pm\,0.0004$	313.78 $\,\pm\,1.67$	47.27 $\,\pm\,0.04$

Appendix

Table 6. Component ablations on Beauty (

N_{q}=4

); notation follows Figure 3 . Mean

\pm

SD over five runs; bold: best accuracy. Time: optimizer-step latency; memory: peak allocated. ∗ : micro-batch 16, accumulation 8.

Compression	Len.	Recall@5	NDCG@5	Recall@10	NDCG@10	Time (ms)	Memory (GB)
Raw input	666.6	0.0711 $\,\pm\,0.0002$	0.0494 $\,\pm\,0.0005$	0.0976 $\,\pm\,0.0004$	0.0579 $\,\pm\,0.0005$	3650.01 $\,\pm\,55.51$ ∗	99.58 $\,\pm\,0.00$ ∗
Mean pooling	31.3	0.0711 $\,\pm\,0.0002$	0.0495 $\,\pm\,0.0002$	0.0970 $\,\pm\,0.0007$	0.0578 $\,\pm\,0.0002$	185.66 $\,\pm\,2.55$	30.58 $\,\pm\,0.00$
Merger, $N_{q}=1$	31.3	0.0726 $\,\pm\,0.0010$	0.0505 $\,\pm\,0.0006$	0.1000 $\,\pm\,0.0008$	0.0593 $\,\pm\,0.0006$	192.34 $\,\pm\,2.57$	31.59 $\,\pm\,0.04$
Merger, $N_{q}=2$	38.5	0.0728 $\,\pm\,0.0008$	0.0509 $\,\pm\,0.0004$	0.1000 $\,\pm\,0.0013$	0.0596 $\,\pm\,0.0005$	227.53 $\,\pm\,2.95$	36.82 $\,\pm\,0.04$
Merger, $N_{q}=4$	52.8	0.0729 $\,\pm\,0.0005$	0.0511 $\,\pm\,0.0006$	0.0998 $\,\pm\,0.0008$	0.0598 $\,\pm\,0.0007$	340.68 $\,\pm\,36.33$	47.27 $\,\pm\,0.04$
Merger, $N_{q}=8$	81.4	0.0724 $\,\pm\,0.0003$	0.0509 $\,\pm\,0.0004$	0.0992 $\,\pm\,0.0009$	0.0596 $\,\pm\,0.0003$	464.41 $\,\pm\,30.91$	68.19 $\,\pm\,0.05$

Appendix

Table 7. Token-budget ablation on Beauty with full supervision. Len.: mean input tokens. Conventions and ∗ follow Table 6 .

Layout	Loss	Recall@5	NDCG@5	Recall@10	NDCG@10
Causal	Uniform	0.0716 $\,\pm\,0.0008$	0.0494 $\,\pm\,0.0006$	0.0993 $\,\pm\,0.0010$	0.0583 $\,\pm\,0.0007$
Tree	Uniform	0.0716 $\,\pm\,0.0004$	0.0492 $\,\pm\,0.0003$	0.0989 $\,\pm\,0.0009$	0.0580 $\,\pm\,0.0005$
Causal	Weighted	0.0719 $\,\pm\,0.0005$	0.0504 $\,\pm\,0.0003$	0.0993 $\,\pm\,0.0012$	0.0592 $\,\pm\,0.0003$
Tree	Weighted	0.0726 $\,\pm\,0.0010$	0.0505 $\,\pm\,0.0006$	0.1000 $\,\pm\,0.0008$	0.0593 $\,\pm\,0.0006$

Appendix

Table 8. Target layout and loss weighting on Beauty (

N_{q}=1

). Mean

\pm

SD over five runs; bold: best results.

Setting	Recall@5	NDCG@5	Recall@10	NDCG@10
Next-item only	0.0497 $\,\pm\,0.0020$	0.0337 $\,\pm\,0.0018$	0.0747 $\,\pm\,0.0027$	0.0417 $\,\pm\,0.0021$
+FB $\times 1$ , $\lambda_{\mathrm{F}}=0.25$	0.0527 $\,\pm\,0.0020$	0.0355 $\,\pm\,0.0014$	0.0786 $\,\pm\,0.0026$	0.0439 $\,\pm\,0.0016$
+FB $\times 1$ , $\lambda_{\mathrm{F}}=0.5$	0.0517 $\,\pm\,0.0028$	0.0348 $\,\pm\,0.0018$	0.0776 $\,\pm\,0.0029$	0.0432 $\,\pm\,0.0018$
+FB $\times 1$ , $\lambda_{\mathrm{F}}=1.0$	0.0485 $\,\pm\,0.0016$	0.0325 $\,\pm\,0.0013$	0.0732 $\,\pm\,0.0016$	0.0404 $\,\pm\,0.0013$
+SB $\times 2$ , $\lambda_{\mathrm{S}}=0.5$	0.0703 $\,\pm\,0.0009$	0.0495 $\,\pm\,0.0007$	0.0976 $\,\pm\,0.0012$	0.0583 $\,\pm\,0.0006$
+SB $\times 2$ , $\lambda_{\mathrm{S}}=1.0$	0.0725 $\,\pm\,0.0006$	0.0510 $\,\pm\,0.0006$	0.0981 $\,\pm\,0.0016$	0.0592 $\,\pm\,0.0008$

Appendix

Table 9. Number and weights of auxiliary branches on Beauty (

N_{q}=1

). Mean

\pm

SD over five runs. FB/SB: future/synthetic branches; shading: selected settings; bold: best results.

Large Language Model-Based Recommendation

Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.

See all 213 papers

REPREC: Representation Driven Parameter-Efficient Recommendation System
Jul 24, 2026Harshini Kavuru, Dwipam Katariya, Giri Iyengar +3Sequential Recommendation Large Language Model-Based Recommendation
The Ohio State University, USA · Capital One, AI Foundations, USA
History-Conditioned Joint-Prefix Alignment for Generative Recommendation
Aug 29, 2026Hongliang Sun, Lianjie Li, Bolin Zhang +3Generative Recommendation Sequential Recommendation
Department of Computer Science, Weihai & Qingdao Research Institute Harbin Institute of Technology Weihai, China
Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction
Oct 11, 2026Sejoon Oh, Fernando Amat Gil, Evan Smothers +11Large Language Model-Based Recommendation Semantic ID Recommendation
Netflix Personalization Foundations Team

Jul 24, 2026J/K move · Enter open · S save

REPREC: Representation Driven Parameter-Efficient Recommendation System

Harshini Kavuru, Dwipam Katariya, Giri Iyengar, Pranab Mohanty+2

The Ohio State University, USA · Capital One, AI Foundations, USA

Sequential Recommendation Large Language Model-Based Recommendation Representation Alignment Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Large language models (LLMs) have been applied to sequential recommendation by formulating it as a natural language task. Previous work has improved personalization by incorporating collaborative and sequential signals through input conditioning or LLM fine-tuning. However, existing approaches often rely on one or more of the following: LLM fine-tuning, additional architectural modules, representation distillation, or item-level conditioning over long interaction histories, increasing training complexity and deployment cost. We propose REPREC, a lightweight framework that reformulates LLM-based sequential recommendation through lightweight user representation alignment. REPREC maps a fixed-size user embedding from a frozen sequential encoder into a small set of learned soft tokens through a lightweight MLP injector that conditions a frozen LLM, leaving both pretrained backbones unchanged while training only the injector. We conducted exhaustive experiments on multiple benchmark datasets and demonstrate that REPREC consistently outperforms LoRA while remaining compatible with different pretrained sequential encoders and LLM backbones, enabling a modular and production-friendly recommendation pipeline without modifying either pretrained component. The gains are particularly pronounced for casual and core users across all datasets, highlighting REPREC's effectiveness in low-data regimes. Finally, when trained on short prompt histories and evaluated with longer contexts, REPREC maintains 85-100% of LoRA's performance while reducing per-epoch training time by an average of 1.51X, demonstrating an effective balance between recommendation quality and computational efficiency for production deployment. The code is available at https://github.com/phdbotcode/REPREC

Abstract

Large Language Model-Based Recommendation

Explore similar work

REPREC: Representation Driven Parameter-Efficient Recommendation System

History-Conditioned Joint-Prefix Alignment for Generative Recommendation

Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction