Decision4Rec: A Jev-Inspired Framework of Decision Models on Knowledge and Active Graphs for Multi-Agent Recommender Systems
Authors: Marco Polignano, Het Darshan Mehta, Pasquale Lops, Ernesto William De Luca, Giovanni Semeraro
Organizations: Università degli studi di Bari Aldo Moro, Piazza Umberto I, 70121 Bari, Italy · Otto-von-Guericke University, Universitätsplatz 2, 39106 Magdeburg, Germany · Leibniz Institute for Educational Media | Georg Eckert Institute, Freisestr. 1, 38118 Braunschweig, Germany
Decision models, such as the commercial Jev, are language models that return one probability for each option of a question instead of generating text. We present Decision4Rec, an open and modular framework that makes decision models useful for recommendation. A knowledge graph writes the logs of each recommendation scenario as short statements on a defined scale, such as "sold: top 10%". A pool of agents, proposes candidates, and a residual ranker keeps the best 30. A decision model, trained once on real choices of three scenarios, reads them through one interface of typed questions, and a fusion step combines its answers with the score of the ranker. Any model with this interface can be used, and we run the commercial Jev, the released open Kev and Kev models trained by us on recommendation requests. The framework with a trained 4B decision model is 25 to 36% above the best trained baseline in NDCG@10 in three sparse Amazon catalogues, one of them never used to train the decision model, and almost tie with baselines on the dense Steam dataset. Most of the gain comes from the pool, and with a trained recommender it comes from what the agents know about its candidates rather than from new candidates. The 27B decision model adds to the pool 5 to 6% above the 0.8B model in three of four domains. A trained 0.8B open model beats the released Jev in eleven of fifteen settings and is never worse. On MovieLens-1M, a dense domain never seen in training, the gain is 5 to 7%. Code, configurations and trained models are available at https://github.com/marcopoli/decision4rec and https://huggingface.co/m-polignano.
Figures & tables
Figure 1: Two ways to use a language model for ranking. A generative model writes a list as text. A decision model returns a probability for each option given by the caller.
Figure 3: The same raw number means different things in different scenarios. Both bars use the same scale of purchases, and the red dot is the same item with 200 purchases. In a small scenario that sells lab supplies it is one of the best sellers (top 10%); in a large video games scenario it is an ordinary item (top half). A statement gives the position of the item inside its scenario. The borders are invented for illustration.
Figure 4: Architecture of Decision4Rec : four modules with fixed contracts (in italics), designed for the ActiveGraph runtime and run as a batch pipeline in the experiments. (1) The knowledge graph of the scenario, never trained, writes statements on scales of the scenario. (2) The multi-agent pool, with the trained recommender as one agent; a residual ranker corrects its score and keeps 30 candidates. (3) The decision model, a slot that any model with the typed-question interface can fill. (4) Fusion with the ranker score.
Figure 5: What each part of the system may see, for one user with history i1,…,iT . Each row is one use of the history; the colored cell is the target. Rule 1: the statistics of the knowledge graph stop before the target, for every user. Rule 2 (not drawn): a record is kept only if its target is among the candidates, and a missing target is never added. Rule 3: when the candidates come from a trained model, the target is an item that this model never learned as a target, so it cannot have memorized the answer. Users of every evaluation case are left out of Decision4Rec -Full’s records.
Figure 6: The runtime. The boxes in the first row are the main events in the log, and the labels above the arrows say what the behaviors do between two events. The dashed row is a counterfactual: the log is forked, one interaction is removed, and the same behaviors run again; the difference between the two lists is the effect of that interaction.
NDCG@10 ↑
VG
MI
Steam
IND
ML-1M
Training-free methods
popularity
0.0126
0.0133
0.0259
0.0088
0.0180
item kNN
0.0516
0.0314
0.0475
0.0231
0.1337
sequential rules
0.0494
0.0287
0.0461
0.0226
0.1425
content kNN
0.0155
0.0111
0.0159
0.0150
0.0277
transitions and text
0.0250
0.0142
0.0274
0.0170
0.0370
Table 6: NDCG@10 of the baselines on all users of each domain (whole catalogue), best configuration of the full grid. Bold: best trained baseline. These values use all users, so they differ a little from the values on the 2,000 development users of Section 7 . MovieLens-1M: the enriched version; with genres only, SASRecF reaches 0.1870 and is the best model.
Figure 7: What the trained decision model adds to the pool, by size. Each bar is the change of the fused NDCG@10 of Decision4Rec -Full against the same pool ordered by its residual ranker alone ( Decision4Rec -Full without decision model), 2,000 development test users per domain. In five of six settings the 4B model adds more than the 0.8B model; on Steam both add 2%. The 27B model, trained on 8,000 of the 34,667 records, adds 4 to 8% in the four domains of Tables 7 and 9 and 1% on both versions of MovieLens-1M. ∗ : significant (p ≤ 0.026); the other differences are not significant on their own. Values from Tables 11 and 10 .
Domain
System
Rare
Middle
Popular
Video Games
users
30%
35%
35%
SASRecF
0.0089
0.0406
0.1292
pool (no decision model)
0.0240∗
0.0544∗
0.1333
Decision4Rec -Full
0.0235∗
0.0615∗
0.1410
Musical Instr.
users
44%
32%
24%
SASRecF
0.0037
0.0195
0.1234
Table 12: Which targets gain: NDCG@10 ↑ on the 2,000 development test users of each domain, split by the purchases of the target item in the training data (rare: fewer than 20; middle: 20 to 99; popular: 100 or more). Decision4Rec -Full: with the 4B decision model, fused. Users: share of the test users in each group. ∗ : significant difference from SASRecF within the group (paired randomization test, p < 0.05). Bold: best system of each group.
Video G.
Musical I.
Steam
Industrial †
ideal
Calibration error, raw ↓
0.019
0.015
0.015
0.031
0
Calibration error, after scaling ↓
0.030
0.025
0.015
0.031
0
Surprise of the next item ↓
3.08
3.17
3.20
3.07
low; uniform: 3.40
Order change ↓
0.0015
0.0013
0.0010
0.0013
0
Position effect
−0.11
−0.10
−0.05
−0.07
0
Conformal sets, coverage
0.957
0.942
0.917
0.932
0.90
Table 13: RQ4. Can the probabilities of the trained Kev-0.8B decision model be trusted? Full architecture, 2,000 test users per domain. The last column gives the ideal value. Calibration error: expected calibration error (ECE) of the probability of the first choice. Surprise of the next item: negative log-likelihood (NLL) of the next item among the 30 options; a uniform guess gives 3.40. After scaling: after temperature scaling on validation users (fitted temperatures 0.95 to 1.20). Order change: mean deviation from a Luce model between the two orders of the same request. Position effect: the largest effect of a position on the log probability, against the first position. Conformal sets: for a target coverage of 90%, on users whose next item is among the 30 options.
Kev-0.8B, trained
Kev-4B, trained
Jev, paid API
Time per user ↓
157–167 ms
565 ms
310–340 ms
Input tokens per request ↓
3.9k–4.4k
3.9k–4.4k
3.6k–5.1k
Hardware
one 24 GB GPU
one 24 GB GPU
none
Money per 1,000 users ↓
GPU and electricity
GPU and electricity
$0.15–0.22
Table 14: RQ4. The cost of one recommendation: one request per user with 30 candidates in two orders. Open models on one RTX 3090; Jev as measured through its API (Section 8 ).