Click-through rate (CTR) prediction is a core component of recommendation and advertising systems. Large language models (LLMs) offer rich semantic priors that improve prediction quality, particularly in cold-start and long-tail settings. Injecting a full-scale LLM into a CTR pipeline is, however, prohibitively expensive at inference time: existing LLM-enhanced CTR methods avoid this cost by caching LLM outputs offline rather than serving the LLM online. We propose a resource-efficient pipeline, SlimFM, that makes online LLM-based semantic modeling practical: (1) dependency-aware structured pruning compresses LLaMA3-8B by 44% to a 4.5B-parameter backbone; (2) LoRA-based field-aware fine-tuning trains the pruned model to produce discriminative field-level semantic embeddings from lightweight textual prompts; (3) a semantic-guided fusion module injects these embeddings into the feature interactions of a conventional CTR model. Across four datasets and six CTR backbones, SlimFM retains over 99% of uncompressed-model accuracy while reducing inference latency by up to 3.4x relative to existing LLM-enhanced baselines.
Figures & tables
Figure 1: The SlimFM framework. The upper branch fine-tunes a pruned LLaMA with feature-field prompts to extract field embeddings, which are projected into pairwise field interactions. The lower branch generates feature embeddings from the CTR model and computes feature interactions. The two streams are fused via element-wise multiplication and aggregated for prediction, with a KL-divergence loss aligning the embedding distributions during training.
Digital Music
MovieLens-1M
Video Games
Gift Cards
#Field
13
8
13
13
#Feature
1.70M
9.00M
60.1M
1.98M
#User
101K
6.04K
2.77M
133K
#Item
70.5K
3.71K
137K
1.14M
#Interact.
130K
100M
4.62M
152K
Density
1.83×10−5
4.47×10−2
1.22×10−5
1.01×10−3
Table 1: Dataset statistics.
FM
DeepFM
FwFM
FmFM
FinalMLP
Wukong
Method
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
Par.
MAC
Lat.
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
(M)
(G)
(ms)
KAR
8032
996
54.62
8032
996
54.71
8032
996
54.78
8032
996
54.81
8033
996
54.95
8034
996
55.08
LLM-CF
7075
829
46.56
7075
829
46.46
7075
829
47.02
7075
829
47.02
7076
829
47.43
7077
829
47.53
SlimFM (Ours)
4504
275
15.99
4504
275
15.98
4504
275
16.89
4504
275
17.58
4504
275
16.41
4505
275
16.42
Table 2: Computational efficiency comparison of LLM-enhanced CTR methods
Gift Cards
Video Games
Digital Music
MovieLens-1M
Mean %
Method
LL ↓
AUC ↑
RelImp
LL ↓
AUC ↑
RelImp
LL ↓
AUC ↑
RelImp
LL ↓
AUC ↑
RelImp
AUC ↑
CELA
0.4658
0.6746
1.02%
0.5798
0.6775
0.44%
0.3364
0.7638
0.37%
0.4031
0.8373
0.24%
0.7383
ClickPrompt
0.4656
0.6754
1.48%
0.5747
0.6799
1.79%
0.3356
0.7652
0.90%
0.4025
0.8379
0.42%
0.7396
Base
0.4606
0.6621
0.5805
0.6673
0.3359
0.7612
0.4081
0.8354
0.7315
KAR
0.4551
0.6643
1.34%
0.5803
0.6705
1.93%
0.3410
0.7607
-0.21%
0.4073
0.8358
0.13%
0.7328
LLM-CF
0.4580
0.6666
2.77%
0.5805
0.6682
0.58%
0.3351
0.7633
0.78%
0.4128
0.8337
-0.50%
0.7330
Table 3: Performance comparisons of SlimFM with existing LLM-enhanced CTR methods across six backbone models and four datasets. SlimFM uses a pruned LLaMA3 (4.5B parameters). LL denotes Logloss and RelImp denotes Relative Improvement. Bold values indicate the best result within each backbone group.
Sparsity
Gift Cards
Video Games
Digital Music
MovieLens-1M
Mean
Recovery
LogLoss
AUC
LogLoss
AUC
LogLoss
AUC
LogLoss
AUC
AUC
Rate
LLaMA3-8B
0.4500
0.6673
0.5791
0.6718
0.3346
0.7629
0.4040
0.8364
0.7346
–
LLaMA3-6B
0.4666
0.6656
0.5798
0.6703
0.3351
0.7625
0.4044
0.8362
0.7337
99.88%
LLaMA3-4.5B
0.4790
0.6644
0.5804
0.6691
0.3357
0.7620
0.4049
0.8360
0.7329
99.77%
LLaMA3-2B
0.4997
0.6623
0.5816
0.6672
0.3359
0.7619
0.4053
0.8357
0.7318
99.62%
Table 4: Sparsity-performance trade-off analysis for SlimFM with LLaMA3 backbones.