MMDM: Measuring Massive Multitask Decision Making
Organizations: The Chinese University of Hong Kong · Hong Kong University of Science and Technology · LIGHTSPEED · Independent Researcher
Abstract
System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together. Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts. It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens. With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks. For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%. These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.
Figures & tables
| Benchmark | Max. options | Context length | New instances | Multi-select | Soft targets | Score tasks |
|---|---|---|---|---|---|---|
| JEVal ( Duan et al., 2026 ) | 255 | 82,678 | ||||
| DecisionBench 1.0 ( Solka, 2026 ) | 255 | 35,435 | ||||
| JevBench ( Standhartinger, 2026 ) | N/A | N/A | ||||
| DecideBench 1.1 ( Yong, 2026 ) | 6 | 229 | ||||
| Jev Evaluation ( Deußer et al., 2026 ) | 151 | N/A | ||||
| ReflexBench 1.0 ( Navarro, 2026 ) | 255 | N/A |
| Decision Model | Single-select Acc. | Multi-select Acc. | SB-F | Soft Brier | Score nMAE |
|---|---|---|---|---|---|
| Heuristic reference | |||||
| Random (uniform) | 25.35 | 3.28 | 0.0000 | 0.2051 | 0.1624 |
| Closed-source decision models | |||||
| Jev 1.14.0 | 67.17 | 22.12 | 0.1187 | 0.1101 | 0.0926 |
| Microsoft Decision-1 | 61.36 | 21.96 | 0.1186 | 0.2795 | 0.1836 |
| Liquid-d1-Ultra | 73.32 | 24.33 | 0.1624 | 0.1321 | 0.0883 |
| Overall | Hard Single-select Accuracy by task category | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Decision Model | ALL | WKN | RDC | DED | OPL | CPL | QTR | WFD | AST | PEX | EVD | CDB | IRT | TSS | APF | SCR | EMR |
| Heuristic reference | |||||||||||||||||
| Random (uniform) | 25.3 | 30.7 | 30.1 | 33.5 | 25.7 | 4.1 | 30.7 | 4.4 | 10.4 | 35.6 | 46.3 | 31.9 | 28.5 | 12.8 | 37.7 | 29.5 | 9.2 |
| Closed-source decision models | |||||||||||||||||
| Jev 1.14.0 | 67.2 | 63.4 | 69.6 | 81.2 | 38.2 | 42.7 | 75.8 | 52.2 | 51.0 | 71.5 | 84.2 | 76.4 | 78.5 | 94.2 | 65.7 | 85.7 | 34.3 |
| Microsoft Decision-1 | 61.4 | 62.6 | 63.5 | 76.8 | 38.2 | 13.5 | 65.0 | 43.5 | 57.8 | 59.0 | 76.2 | 76.4 | 60.4 | 79.5 | 66.1 | 71.4 | 47.8 |
| Model | Selected option |
|---|---|
| Perplexity V1.2 | Invalid (0.353) |
| Jev | Valid (0.520) |
| Microsoft Decision-1 | Valid (0.894) |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Capability and example |
|---|---|
| WKN World knowledge | Apply physical, social, scientific, and general knowledge to a situated request. Options distinguish plausible causes, intentions, continuations, or facts. Example F . |
| RDC Reading comprehension | Identify and connect evidence within passages, long documents, or multiple documents. Includes answerability, cross-paragraph inference, and multi-hop evidence composition. Example F . |
| DED Deductive reasoning | Derive what follows from stated premises and rules, including entailment, quantifier scope, and natural-language logical arguments. Example F . |
| OPL Option logic | Resolve propositions that refer to one another inside option criteria. Select either the unique complete interpretation or every true proposition under a consistent assignment. Example F . |
| CPL Constraint planning | Identify complete plans satisfying interacting eligibility, resource, scheduling, and precedence constraints. Several plans can be feasible. Example F . |
| QTR Quantitative and temporal reasoning | Apply practical arithmetic, table and financial evidence, temporal ordering, numeric thresholds, and freshness requirements to a decision. Example F . |
| Source | Items | Categories | Adaptation role |
|---|---|---|---|
| Adaptive-RAG ( Jeong et al., 2024 ) | 267 | IRT | Route to a retrieval strategy |
| AG News ( Zhang et al., 2015 ) | 18 | IRT | Topic or intent mapped to named options |
| ARC ( Clark et al., 2018 ) | 3 | WKN | Knowledge and contextual answer selection |
| BANKING77 ( Casanueva et al., 2020 ) | 2 | IRT | Topic or intent mapped to named options |
| BFCL V4 ( Patil et al., 2025 ) | 4 | TSS | Tool names and contracts as options |
| BIRD Mini-Dev ( Li et al., 2023 ) | 80 | CDB | SQL decisions with database evidence |
| Source | Items | Categories | Adaptation role |
|---|---|---|---|
| RewardBench 2 ( Malik et al., 2026 ) | 68 | APF | Compare responses under source criteria |
| RM-Bench ( Liu et al., 2025 ) | 59 | APF | Compare responses under source criteria |
| RuleTaker ( Clark et al., 2020 ) | 73 | DED | Premise–claim or rule-based judgments |
| SciQ ( Welbl et al., 2017 ) | 12 | WKN | Knowledge and contextual answer selection |
| SkillsBench ( Li et al., 2026 ) | 18 | TSS | Task/environment context and skill descriptions |
| SNLI ( Bowman et al., 2015 ) | 57 | DED | Premise–claim or rule-based judgments |
| Construction family | Items | Mechanism and reference derivation |
|---|---|---|
| Scenario-based decision constructions | 260 | Concrete policy, evidence, tool, planning, and scoring scenarios; explicit finite outcome models for soft targets. |
| Compositional rule and evidence programs | 157 | Thresholds, exceptions, versions, evidence joins, and output constraints; computed targets and verified natural-language renderings. |
| Single-select logical interpretations | 152 | Enumerate assignments to mutually referring propositions; each option gives a complete interpretation. |
| Interdependent reports | 359 | Enumerate the consistent assignment and select its true propositions; cross-check with an independent truth-table implementation. |
| Constraint-based plans | 240 | Enumerate proposed schedules and resource assignments; evaluate every feasibility condition. |
| Version-sensitive workflows | 250 | Simulate prerequisites, edits, certificates, and action order; independently verify positions and version requirements. |
| Decision Model and release reference | Context budget | Cap | Counter | Native | Grouped |
|---|---|---|---|---|---|
| Autotrust JEV 27B-VL ( AutoTrust AI Lab, 2026 ) | 65536 | 256 | provider | 5380 | 68 |
| Clef ( Cloudflare, 2026 ) | 65536 | 255 | proxy | 5354 | 94 |
| Clef-Flash ( Cloudflare, 2026 ) | 24576 | 255 | proxy | 5422 | 26 |
| Clef-Omni ( Cloudflare, 2026 ) | 65536 | 255 | proxy | 5354 | 94 |
| Decisio-Gemma-4-12B ( Gemma Team, 2026 ; Roudaki, 2026 ) | 32768 | 255 | provider | 5396 | 52 |
| Decisio-Gemma-4-31B ( Gemma Team, 2026 ; Roudaki, 2026 ) | 32768 | 255 | provider | 5396 | 52 |
| Hard Multi-select Accuracy (%, ) | Soft Brier ( ) | ||||||||||
| Decision Model | RDC | OPL | CPL | WFD | TSS | CPL | QTR | PEX | EVD | TSS | SCR |
| Heuristic reference | |||||||||||
| Random (uniform) | 6.67 | 4.40 | 0.61 | 0.59 | 3.71 | 0.0026 | 0.0821 | 0.0334 | 0.1039 | 0.0816 | 0.2206 |
| Closed-source decision models | |||||||||||
| Jev 1.14.0 | 100.00 | 5.01 | 22.50 | 21.35 | 80.58 | 0.0425 | 0.0453 | 0.0374 | 0.0275 | 0.0937 | 0.1170 |
| Microsoft Decision-1 | 50.00 | 10.86 | 8.75 | 16.85 | 74.76 | 0.2496 | 0.1475 | 0.1133 | 0.1371 | 0.1647 | 0.2959 |
| Decision Model | Hard Single-select Acc. | Hard Multi-select Acc. | Soft Brier | Score nMAE | SB-F | ECE |
|---|---|---|---|---|---|---|
| Perplexity V1.2 | 76.66 | 36.90 | 0.1694 | 0.1641 | 0.1990 | 0.046 |
| Jev 1.14.0 | 68.73 | 22.75 | 0.1101 | 0.0927 | 0.1198 | 0.054 |
| Microsoft Decision-1 | 64.27 | 24.67 | 0.2794 | 0.1836 | 0.1260 | 0.125 |
| Intern-Decision 4B | 51.70 | 11.85 | 0.1565 | 0.1450 | 0.0845 | 0.085 |
| OpenJev 4B | 61.82 | 18.93 | 0.3685 | 0.3105 | 0.1183 | 0.092 |
| Decision Model | OPL | CPL | WFD | TSS |
|---|---|---|---|---|
| Perplexity V1.2 | 19.78 | 25.00 | 37.08 | 85.44 |
| Jev 1.14.0 | 5.01 | 22.50 | 21.35 | 80.58 |
| Microsoft Decision-1 | 10.86 | 8.75 | 16.85 | 74.76 |
| Intern-Decision 4B | 4.46 | 5.00 | 4.49 | 43.69 |
| OpenJev 4B | 5.29 | 18.75 | 12.36 | 63.11 |
| Native | Grouped | |||
|---|---|---|---|---|
| Decision Model | Items | Acc. | Items | Acc. |
| Perplexity V1.2 | 605 | 35.04 | 28 | 7.14 |
| Jev 1.14.0 | 605 | 22.81 | 28 | 7.14 |
| Microsoft Decision-1 | 611 | 22.75 | 22 | 0.00 |
| Intern-Decision 4B | 541 | 12.75 | 92 | 1.09 |
| OpenJev 4B | 633 | 17.38 | 0 | – |
| Overall | SB-F by category | ||||||||
| Decision Model | Rank | SB-F | OPL | CPL | WFD | TSS | |||
| Heuristic reference | |||||||||
| Random (uniform) | 633 | — | 0.0000 | 0.000 | 1.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Closed-source decision models | |||||||||
| Jev 1.14.0 | 605 | 11 | 0.1057 | 0.137 | 0.915 | 0.022 | 0.244 | 0.261 | 0.465 |
| Microsoft Decision-1 | 601 | 8 | 0.1194 | 0.169 | 0.812 | 0.069 | 0.186 | 0.212 | 0.339 |
| Decision Model | Valid top choice (%) | Multi-select Acc. (%) | Coverage (%) | Narrow margins (count) |
|---|---|---|---|---|
| Random (uniform) | 51.7 | 4.2 | 0.0 | — |
| WeMM-JEV 27B | 88.7 | 38.7 | 64.7 | 43/110 |
| Perplexity Decider-V1.2 | 84.9 | 34.9 | 59.8 | 30/99 |
| Jev 1.14.0 | 63.7 | 20.1 | 38.2 | 14/57 |
| Microsoft Decision-1 | 71.5 | 23.2 | 45.8 | 24/66 |
| GPT-6 Luna Decisions | 64.8 | 12.7 | 34.8 | 20/36 |
| Task | Condition | Tokens | Perplexity V1.2 | Jev 1.14.0 | |
|---|---|---|---|---|---|
| Reading Comprehension, four options, complete state | |||||
| Long-document | 16–32K tokens | 49 | 20,393 | 61.2 | 59.2 |
| Long-document | 32–65K tokens | 16 | 40,166 | 62.5 | 68.8 |
| Multi-hop | 16–32K tokens | 19 | 24,520 | 57.9 | 52.6 |
| Multi-hop | 32–65K tokens | 32 | 51,776 | 50.0 | 46.9 |
| Large catalogs, complete state | |||||
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 279 papers