Click-through rate (CTR) prediction is a core component of recommendation and advertising systems. Large language models (LLMs) offer rich semantic priors that improve prediction quality, particularly in cold-start and long-tail settings. Injecting a full-scale LLM into a CTR pipeline is, however, prohibitively expensive at inference time: existing LLM-enhanced CTR methods avoid this cost by caching LLM outputs offline rather than serving the LLM online. We propose a resource-efficient pipeline, SlimFM, that makes online LLM-based semantic modeling practical: (1) dependency-aware structured pruning compresses LLaMA3-8B by 44% to a 4.5B-parameter backbone; (2) LoRA-based field-aware fine-tuning trains the pruned model to produce discriminative field-level semantic embeddings from lightweight textual prompts; (3) a semantic-guided fusion module injects these embeddings into the feature interactions of a conventional CTR model. Across four datasets and six CTR backbones, SlimFM retains over 99% of uncompressed-model accuracy while reducing inference latency by up to 3.4x relative to existing LLM-enhanced baselines.
Figures & tables
Figure 1: The SlimFM framework. The upper branch fine-tunes a pruned LLaMA with feature-field prompts to extract field embeddings, which are projected into pairwise field interactions. The lower branch generates feature embeddings from the CTR model and computes feature interactions. The two streams are fused via element-wise multiplication and aggregated for prediction, with a KL-divergence loss aligning the embedding distributions during training.
Digital Music
MovieLens-1M
Video Games
Gift Cards
#Field
13
8
13
13
#Feature
1.70M
9.00M
60.1M
1.98M
#User
101K
6.04K
2.77M
133K
#Item
70.5K
3.71K
137K
1.14M
#Interact.
130K
100M
4.62M
152K
Density
1.83×10−5
4.47×10−2
1.22×10−5
1.01×10−3
Table 1: Dataset statistics.
FM
DeepFM
FwFM
FmFM
FinalMLP
Wukong
Method
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
KAR
8032
996
54.62
8032
996
54.71
8032
996
54.78
8032
996
54.81
8033
996
54.95
8034
996
55.08
LLM-CF
7075
829
46.56
7075
829
46.46
7075
829
47.02
7075
829
47.02
7076
829
47.43
7077
829
47.53
SlimFM (Ours)
4504
275
15.99
4504
275
15.98
4504
275
16.89
4504
275
17.58
4504
275
16.41
4505
275
16.42
Table 2: Computational efficiency comparison of LLM-enhanced CTR methods
Gift Cards
Video Games
Digital Music
MovieLens-1M
Mean %
Method
LL ↓
AUC ↑
RelImp
LL ↓
AUC ↑
RelImp
LL ↓
AUC ↑
RelImp
LL ↓
AUC ↑
RelImp
AUC ↑
CELA
0.4658
0.6746
1.02%
0.5798
0.6775
0.44%
0.3364
0.7638
0.37%
0.4031
0.8373
0.24%
0.7383
ClickPrompt
0.4656
0.6754
1.48%
0.5747
0.6799
1.79%
0.3356
0.7652
0.90%
0.4025
0.8379
0.42%
0.7396
Base
0.4606
0.6621
0.5805
0.6673
0.3359
0.7612
0.4081
0.8354
0.7315
KAR
0.4551
0.6643
1.34%
0.5803
0.6705
1.93%
0.3410
0.7607
-0.21%
0.4073
0.8358
0.13%
0.7328
LLM-CF
0.4580
0.6666
2.77%
0.5805
0.6682
0.58%
0.3351
0.7633
0.78%
0.4128
0.8337
-0.50%
0.7330
Table 3: Performance comparisons of SlimFM with existing LLM-enhanced CTR methods across six backbone models and four datasets. SlimFM uses a pruned LLaMA3 (4.5B parameters). LL denotes Logloss and RelImp denotes Relative Improvement. Bold values indicate the best result within each backbone group.
Sparsity
Gift Cards
Video Games
Digital Music
MovieLens-1M
Mean
Recovery
LogLoss
AUC
LogLoss
AUC
LogLoss
AUC
LogLoss
AUC
AUC
Rate
LLaMA3-8B
0.4500
0.6673
0.5791
0.6718
0.3346
0.7629
0.4040
0.8364
0.7346
–
LLaMA3-6B
0.4666
0.6656
0.5798
0.6703
0.3351
0.7625
0.4044
0.8362
0.7337
99.88%
LLaMA3-4.5B
0.4790
0.6644
0.5804
0.6691
0.3357
0.7620
0.4049
0.8360
0.7329
99.77%
LLaMA3-2B
0.4997
0.6623
0.5816
0.6672
0.3359
0.7619
0.4053
0.8357
0.7318
99.62%
Table 4: Sparsity-performance trade-off analysis for SlimFM with LLaMA3 backbones.
On online advertising platforms, newly introduced promotional ads face the cold-start problem, as they lack sufficient user feedback for model training. In this work, we propose LLM-HYPER, a novel framework that treats large language models (LLMs) as hypernetworks to directly generate the parameters of the click-through rate (CTR) estimator in a training-free manner. LLM-HYPER uses few-shot Chain-of-Thought prompting over multimodal ad content (text and images) to infer feature-wise model weights for a linear CTR predictor. By retrieving semantically similar past campaigns via CLIP embeddings and formatting them into prompt-based demonstrations, the LLM learns to reason about customer intent, feature influence, and content relevance. To ensure numerical stability and serviceability, we introduce normalization and calibration techniques that align the generated weights with production-ready CTR distributions. Extensive offline experiments show that LLM-HYPER significantly outperforms cold-start baselines in NDCG@10 by 55.9%. Our real-world online A/B test on one of the top e-commerce platforms in the U.S. demonstrates the strong performance of LLM-HYPER, which drastically reduces the cold-start period and achieves competitive performance. LLM-HYPER has been successfully deployed in production.
LLM-based recommender systems increasingly generate chain-of-thought (CoT) text before each item prediction to activate the LLM's reasoning capacity, costing tens to hundreds of extra tokens per recommendation at inference. We ask whether inference-time reasoning artifacts (textual or latent) can be replaced by training-time supervision alone, without loss of accuracy. We propose LatentRec, a framework for LLM-based generative recommendation that replaces inference-time reasoning artifacts with training-time per-step head supervision. LatentRec has three components. A 2.5M-parameter Continuous Thought Module iterates K latent reasoning steps over the LLM's hidden state during training. Soft Token Selection bridges the ~14x norm gap between hidden states and input embeddings. A Per-step Reasoning Loss (PRL) supervises every intermediate state through the LLM's prediction head against the first token of the target Semantic ID. The trained LLM retains this accuracy without the module at inference, so the module can be discarded at deployment. On three Amazon benchmarks, LatentRec improves HR@10 by 4.0-5.0% (relative) over a CoT-distilled baseline at 18.3x lower latency (batch=1). Per-step training supervision consistently improves accuracy across three datasets, paid once at training rather than at every recommendation.
Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks instead of developing custom machine learning models. LLMs require few, if any, training examples and can be utilized by users without expertise in model development. However, this comes at the cost of substantially higher resource and energy consumption compared to smaller models, which often achieve similar predictive performance for simple tasks. In this paper, we present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task. JITR retains the ease of use and low development effort of LLMs, while saving significant cost and energy. We discuss the main challenges in realizing our vision regarding the identification of recurring tasks and the creation of a custom model. Specifically, we argue that model search and transfer learning will play a crucial role in JITR to efficiently identify and fine-tune models for a recurring task. Using our JITR prototype Poodle, we reduce inference time by up to 7.5x compared to a self- hosted LLM and save more than $2,200 per 1M requests compared to a flagship hosted LLM, while achieving accuracy competitive with the LLM baseline.
Nils Strassenburg, Boris Glavic, Tilmann Rabl
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago