Decision models such as Jev return probabilities over caller-defined options and are increasingly used for classification. Large label spaces challenge both accuracy and scalability: larger menus can impair decisions, increase input costs, and exceed option limits. Hierarchical classification reduces menu size but introduces serial calls and error propagation. We propose AdaMenu, which builds a compact candidate menu from retrieved labeled examples and makes a single decision-model request. Its candidate labels adapt to the query, and its default setting also adjusts menu size using aggregated retrieval scores. We evaluate AdaMenu with open and closed decision models on four label sets containing up to 1,016 labels. It outperforms TypeSafe's beam-search baseline on all 16 model and label-set pairs where both methods run, improving accuracy by 16.6-34.3 percentage points. Relative to the full menu on the three smaller label sets, accuracy differences range from -0.5 to +7.2 percentage points, while input token usage falls by factors of 3.2-22.9. AdaMenu also runs when option limits exclude the full menu, including settings where the hierarchical baseline is unavailable.
Figures & tables
Figure 1: Large menus challenge accuracy and scalability. (a) A small menu keeps input cost low. (b) More options can reduce accuracy and increase input cost, even within the option limit. (c) Menus exceeding the limit are rejected.
Figure 2: Greedy label-tree traversal versus AdaMenu, on the query of Figure 1 . Left: successive decisions along the tree can exclude the correct branch early. Right: retrieval constructs an adaptive candidate menu (partially shown), followed by a single decision-model request.
Model
One read
Most
Longer menus
Jev-1.14 (closed)
–
255
rejected
Jeff-1.1-Qwen-4B
16
256
groups + final read
StartLux-4B
25
any
knock-out
Eikos-4B
100
any
tournament
Eikos-27B
160
any
tournament
Jeff-1.0-Large
16
16
rejected
Table 1: The decision models and how they read a menu: options in one forward pass (One read), most options in one question (Most), and what happens to a longer menu. Jev-1.14 is hosted; the open models are served by us.
Data
Queries
Labels
Depth
Fan-out
BANKING77
1,033
77
–
–
CLINC150
3,000
150
2
15
DBpedia
5,000
219
3
30
O*NET
5,329
1,016
5
23
Table 2: Label sets. Depth: levels of the label tree. Fan-out: the most children of any node, i.e., the largest menu used by the hierarchical baseline.
BANKING77
CLINC150
DBpedia
O*NET
Model
Full
Split
AdaMenu
Full
Split
Tree
AdaMenu
Full
Split
Tree
AdaMenu
Full
Split
Tree
AdaMenu
Jev-1.14
79.3
–
80.1
92.3
–
69.9
93.0
93.0
–
75.1
93.3
–
–
44.4
70.1
Jeff-1.1-Qwen-4B
–
78.3
80.8
–
90.8
68.8
92.2
–
91.8
72.0
93.1
–
–
37.9
66.2
StartLux-4B
–
89.2
89.6
–
96.3
79.2
95.8
–
91.5
73.1
92.8
–
69.5
38.2
67.4
Eikos-4B
70.8
–
76.4
–
81.7
60.0
88.6
–
83.9
56.8
91.0
–
57.2
29.7
62.0
Eikos-27B
77.1
–
79.6
92.7
–
65.8
92.2
–
91.5
71.2
93.2
–
68.0
40.1
66.9
Table 3: Accuracy (%) of the full menu, the hierarchical baseline (Tree, beam 3) and AdaMenu; best per model and label set in bold. Full: the model reads all labels in one pass. Split: its serving code splits them into groups read in several passes, within one request. –: not available for the model. AdaMenu uses adaptive menu sizes, except K=10 for Jeff-1.0-Large. Top-1 kNN candidate: AdaMenu without the model request (an ablation). Every method and model: Table 6 .
Figure 3: Accuracy versus input tokens per query (log-scale x-axis). Points show medians over the models available for each method; vertical bars show the interquartile range of accuracy.
Full
Tree
AdaMenu
Data
Tokens
USD
Tokens
USD
Tokens
USD
BANKING77
1,621
0.068
–
–
508
0.021
CLINC150
2,438
0.102
1,432
0.060
441
0.019
DBpedia
3,823
0.161
2,255
0.095
514
0.022
O*NET
–
–
3,160
0.133
804
0.034
Table 4: Cost of Jev-1.14 per method: Tokens are input tokens per query (all requests); USD is the price per 1,000 queries. Tree: the hierarchical baseline with beam 3. –: no supplied label tree (BANKING77) or menu rejected (O*NET).
Figure 4: Per-model accuracy of the hierarchical baseline (beam 3), AdaMenu and, where the model accepts it, the full menu. n/a: the tree has nodes with more children than Jeff-1.0-Large reads.
CLINC150
DBpedia
O*NET
Model
Full
Tree
leaf ∣ top
AdaMenu
Full
Tree
leaf ∣ top
AdaMenu
Full
Tree
leaf ∣ top
AdaMenu
Jev-1.14
97.4
70.1
95.8
97.9
98.6
90.2
82.1
98.7
–
69.9
62.7
84.0
Jeff-1.1-Qwen-4B
96.9
65.2
95.2
97.7
98.2
85.5
82.3
98.6
–
64.1
57.6
81.6
StartLux-4B
99.0
76.4
98.7
98.8
98.4
88.5
80.8
98.7
84.5
64.8
55.7
82.6
Eikos-4B
92.6
59.4
90.9
96.5
95.5
71.8
78.4
97.9
78.4
58.4
50.1
79.6
Eikos-27B
98.2
64.5
94.3
97.8
97.8
86.9
82.7
98.6
84.5
68.8
58.1
82.8
Table 5: First-step routing per model (%). Full (the full menu, read in one pass or split) / AdaMenu: share of the method’s predictions under the gold label’s top-level node. Tree: share of queries whose greedy first step picks that node. leaf ∣ top : accuracy of greedy descent among the queries it routed correctly. –: menu not accepted.
Data
Model
Full
Split
Greedy
Beam-3
K=5
K=10
K=20
τ
BANKING77
Coverage
96.0
98.1
99.3
99.0
Jev-1.14
79.3
–
–
–
80.9
80.3
80.4
80.1
Jeff-1.1-Qwen-4B
–
78.3
–
–
80.4
80.5
78.2
80.8
StartLux-4B
–
89.2
–
–
87.8
88.9
89.7
89.6
Eikos-4B
70.8
–
–
–
76.6
76.1
73.1
76.4
Eikos-27B
77.1
–
–
–
78.7
79.0
76.6
79.6
Table 6: Accuracy (%) of every method and model. Full, Split: all labels in one request, read in one pass (Full) or split by the model’s serving code into groups read in several passes (Split). Greedy, Beam-3: variants of the hierarchical baseline. K , τ : AdaMenu with K∈{5,10,20} candidates or adaptive. Coverage : share of queries whose gold label is among the candidates of each AdaMenu setting. –: not available (no supplied label tree, or more options than the model accepts).
AdaMenu accuracy
Data
Bank
No ex.
Cov.
Top-1
Jev
Jeff-1.1
StartLux
Eikos-4B
Eikos-27B
Jeff-1.0-L
BANKING77
full
0
99.0
78.7
80.1
80.8
89.6
76.4
79.6
80.4
10%
0
97.2
55.2
78.6
77.8
88.2
73.0
77.2
76.0
10%, strat.
0
97.3
61.0
79.3
78.2
87.7
71.3
77.1
76.3
CLINC150
full
0
98.2
80.6
93.0
92.2
95.8
88.6
92.2
92.8
10%
0
94.2
60.3
88.9
87.8
92.1
83.9
87.7
87.2
Table 7: AdaMenu with the full bank, a hash-selected tenth and a stratified tenth (every label with an example in the full bank keeps at least one). No ex.: labels without any example. Cov.: coverage of the adaptive candidates (%). Top-1: accuracy of the top-1 kNN candidate (%). AdaMenu accuracy (%): adaptive menu sizes, except K=10 for Jeff-1.0-Large.