Decision4Rec: A Jev-Inspired Framework of Decision Models on Knowledge and Active Graphs for Multi-Agent Recommender Systems
Authors: Marco Polignano, Het Darshan Mehta, Pasquale Lops, Ernesto William De Luca, Giovanni Semeraro
Organizations: Università degli studi di Bari Aldo Moro, Piazza Umberto I, 70121 Bari, Italy · Otto-von-Guericke University, Universitätsplatz 2, 39106 Magdeburg, Germany · Leibniz Institute for Educational Media | Georg Eckert Institute, Freisestr. 1, 38118 Braunschweig, Germany
Decision models, such as the commercial Jev, are language models that return one probability for each option of a question instead of generating text. We present Decision4Rec, an open and modular framework that makes decision models useful for recommendation. A knowledge graph writes the logs of each recommendation scenario as short statements on a defined scale, such as "sold: top 10%". A pool of agents, proposes candidates, and a residual ranker keeps the best 30. A decision model, trained once on real choices of three scenarios, reads them through one interface of typed questions, and a fusion step combines its answers with the score of the ranker. Any model with this interface can be used, and we run the commercial Jev, the released open Kev and Kev models trained by us on recommendation requests. The framework with a trained 4B decision model is 25 to 36% above the best trained baseline in NDCG@10 in three sparse Amazon catalogues, one of them never used to train the decision model, and almost tie with baselines on the dense Steam dataset. Most of the gain comes from the pool, and with a trained recommender it comes from what the agents know about its candidates rather than from new candidates. The 27B decision model adds to the pool 5 to 6% above the 0.8B model in three of four domains. A trained 0.8B open model beats the released Jev in eleven of fifteen settings and is never worse. On MovieLens-1M, a dense domain never seen in training, the gain is 5 to 7%. Code, configurations and trained models are available at https://github.com/marcopoli/decision4rec and https://huggingface.co/m-polignano.
Figures & tables
Figure 1: Two ways to use a language model for ranking. A generative model writes a list as text. A decision model returns a probability for each option given by the caller.
Figure 3: The same raw number means different things in different scenarios. Both bars use the same scale of purchases, and the red dot is the same item with 200 purchases. In a small scenario that sells lab supplies it is one of the best sellers (top 10%); in a large video games scenario it is an ordinary item (top half). A statement gives the position of the item inside its scenario. The borders are invented for illustration.
Figure 4: Architecture of Decision4Rec : four modules with fixed contracts (in italics), designed for the ActiveGraph runtime and run as a batch pipeline in the experiments. (1) The knowledge graph of the scenario, never trained, writes statements on scales of the scenario. (2) The multi-agent pool, with the trained recommender as one agent; a residual ranker corrects its score and keeps 30 candidates. (3) The decision model, a slot that any model with the typed-question interface can fill. (4) Fusion with the ranker score.
Figure 5: What each part of the system may see, for one user with history i1,…,iT . Each row is one use of the history; the colored cell is the target. Rule 1: the statistics of the knowledge graph stop before the target, for every user. Rule 2 (not drawn): a record is kept only if its target is among the candidates, and a missing target is never added. Rule 3: when the candidates come from a trained model, the target is an item that this model never learned as a target, so it cannot have memorized the answer. Users of every evaluation case are left out of Decision4Rec -Full’s records.
Figure 6: The runtime. The boxes in the first row are the main events in the log, and the labels above the arrows say what the behaviors do between two events. The dashed row is a counterfactual: the log is forked, one interaction is removed, and the same behaviors run again; the difference between the two lists is the effect of that interaction.
NDCG@10 ↑
VG
MI
Steam
IND
ML-1M
Training-free methods
popularity
0.0126
0.0133
0.0259
0.0088
0.0180
item kNN
0.0516
0.0314
0.0475
0.0231
0.1337
sequential rules
0.0494
0.0287
0.0461
0.0226
0.1425
content kNN
0.0155
0.0111
0.0159
0.0150
0.0277
transitions and text
0.0250
0.0142
0.0274
0.0170
0.0370
Table 6: NDCG@10 of the baselines on all users of each domain (whole catalogue), best configuration of the full grid. Bold: best trained baseline. These values use all users, so they differ a little from the values on the 2,000 development users of Section 7 . MovieLens-1M: the enriched version; with genres only, SASRecF reaches 0.1870 and is the best model.
Figure 7: What the trained decision model adds to the pool, by size. Each bar is the change of the fused NDCG@10 of Decision4Rec -Full against the same pool ordered by its residual ranker alone ( Decision4Rec -Full without decision model), 2,000 development test users per domain. In five of six settings the 4B model adds more than the 0.8B model; on Steam both add 2%. The 27B model, trained on 8,000 of the 34,667 records, adds 4 to 8% in the four domains of Tables 7 and 9 and 1% on both versions of MovieLens-1M. ∗ : significant (p ≤ 0.026); the other differences are not significant on their own. Values from Tables 11 and 10 .
Domain
System
Rare
Middle
Popular
Video Games
users
30%
35%
35%
SASRecF
0.0089
0.0406
0.1292
pool (no decision model)
0.0240∗
0.0544∗
0.1333
Decision4Rec -Full
0.0235∗
0.0615∗
0.1410
Musical Instr.
users
44%
32%
24%
SASRecF
0.0037
0.0195
0.1234
Table 12: Which targets gain: NDCG@10 ↑ on the 2,000 development test users of each domain, split by the purchases of the target item in the training data (rare: fewer than 20; middle: 20 to 99; popular: 100 or more). Decision4Rec -Full: with the 4B decision model, fused. Users: share of the test users in each group. ∗ : significant difference from SASRecF within the group (paired randomization test, p < 0.05). Bold: best system of each group.
Video G.
Musical I.
Steam
Industrial †
ideal
Calibration error, raw ↓
0.019
0.015
0.015
0.031
0
Calibration error, after scaling ↓
0.030
0.025
0.015
0.031
0
Surprise of the next item ↓
3.08
3.17
3.20
3.07
low; uniform: 3.40
Order change ↓
0.0015
0.0013
0.0010
0.0013
0
Position effect
−0.11
−0.10
−0.05
−0.07
0
Conformal sets, coverage
0.957
0.942
0.917
0.932
0.90
Table 13: RQ4. Can the probabilities of the trained Kev-0.8B decision model be trusted? Full architecture, 2,000 test users per domain. The last column gives the ideal value. Calibration error: expected calibration error (ECE) of the probability of the first choice. Surprise of the next item: negative log-likelihood (NLL) of the next item among the 30 options; a uniform guess gives 3.40. After scaling: after temperature scaling on validation users (fitted temperatures 0.95 to 1.20). Order change: mean deviation from a Luce model between the two orders of the same request. Position effect: the largest effect of a position on the log probability, against the first position. Conformal sets: for a target coverage of 90%, on users whose next item is among the 30 options.
Kev-0.8B, trained
Kev-4B, trained
Jev, paid API
Time per user ↓
157–167 ms
565 ms
310–340 ms
Input tokens per request ↓
3.9k–4.4k
3.9k–4.4k
3.6k–5.1k
Hardware
one 24 GB GPU
one 24 GB GPU
none
Money per 1,000 users ↓
GPU and electricity
GPU and electricity
$0.15–0.22
Table 14: RQ4. The cost of one recommendation: one request per user with 30 candidates in two orders. Open models on one RTX 3090; Jev as measured through its API (Section 8 ).
Large language models (LLMs) have shown promise for recommendation reranking, but their use introduces an important trade-off between recommendation quality and serving efficiency. We investigate whether decision-oriented systems provide a useful alternative when the reranking task is fundamentally a structured choice among predefined candidate items. Specifically, we conduct a controlled empirical study of two decision-oriented systems, TypeSafe Jev and OpenAI Decisions, for recommendation reranking. Across three Amazon Reviews domains and four candidate-set sizes, we compare these systems against recommendation-specific models and pointwise and listwise LLM-based rerankers, systematically evaluating recommendation effectiveness, candidate-order robustness, and efficiency and scalability. Our results reveal that Jev frequently achieves a favorable quality--latency trade-off, offering competitive recommendation effectiveness at substantially lower observed latency than stronger LLM-based alternatives. Overall, our findings suggest that specialization for decision-making does not inherently guarantee superior recommendation quality or robustness. Instead, the practical value of decision-oriented inference depends on implementation-specific characteristics and application requirements. These findings motivate further investigation of decision-oriented systems for recommendation and other ranking tasks with structured output spaces.
Open-ended search combined with foundation models has produced advances such as faster algorithms, new results on open mathematical problems, and self-improving agents. We extend this paradigm to recommendation for decision-making, focusing on decisions where the set of possible choices is fixed but the reasons for choosing them are not. In high-stakes decisions, e.g., which company to acquire or which customer to target, comparing diverse, well-reasoned options can be more useful than a single ranked list. We formalize open-ended recommendation as building a diverse, high-quality set of recommendations, each an item together with a hypothesis, a natural-language reason why the item fits the needs of a decision maker. We propose QD-Rec, a quality--diversity method that selects items with a contextual bandit and generates hypotheses with a foundation model from earlier ones kept in an archive as stepping stones. A new recommendation enters the archive if it is novel or better than its nearest neighbor. Across M&A, customer targeting, and paper and book recommendation, QD-Rec gives the highest mean quality and builds the largest archive in most settings.
General decision models, such as Jev, have recently emerged as efficient alternatives to LLMs for structured judgment and selection. But what kinds of decisions can these models reliably make, and how does their behavior change when individual decisions are composed into larger systems? To study this, we introduce JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, and evaluate 25 model configurations spanning general decision models and generative LLMs. Our results show that (1) general decision models are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation: they can often identify the most likely outcome while substantially overstating its probability. (2) In more dynamic and realistic systems involving long-horizon, multi-step interactions, the advantages of fast local decision making are offset by reliability failures at the system level. on τ-bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories. (3) In large-scale social simulation, decision models approach strong generative LLMs on individual response prediction at substantially lower inference cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias. Finally, we propose SII-Decision-4B and SII-Decision-27B, which internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation, with SII-Decision-27B performing on par with Jev on JEVal while answering a typical query in about 0.1 s.
Feiyu Duan, Jiayu Lin, Jia Wang +6
Shanghai Innovation Institute · Fudan University · Tongji University +2