CONDESION-BENCH: Conditional Decision-Making of Large Language Models in Compositional Action Space

Yeonjun Hwang; Sungyong Park; Minju Kim; Dongha Lee; Jinyoung Yeo

cs.CLApr 10, 2026

CONDESION-BENCH: Conditional Decision-Making of Large Language Models in Compositional Action Space

Authors: Yeonjun Hwang, Sungyong Park, Minju Kim, Dongha Lee, Jinyoung Yeo

Organizations: Department of Artificial Intelligence, Yonsei University

Abstract

Large language models have been widely explored as decision-support tools in high-stakes domains due to their contextual understanding and reasoning capabilities. However, existing decision-making benchmarks rely on two simplifying assumptions: actions are selected from a finite set of pre-defined candidates, and explicit conditions restricting action feasibility are not incorporated into the decision-making process. These assumptions fail to capture the compositional structure of real-world actions and the explicit conditions that constrain their validity. To address these limitations, we introduce CONDESION-BENCH, a benchmark designed to evaluate conditional decision-making in compositional action space. In CONDESION-BENCH, actions are defined as allocations to decision variables and are restricted by explicit conditions at the variable, contextual, and allocation levels. By employing oracle-based evaluation of both decision quality and condition adherence, we provide a more rigorous assessment of LLMs as decision-support tools.

Figures & tables

Figure 1: A comparison between conventional decision-making benchmarks (top) and Condesion-Bench (bottom). While prior benchmarks focus on selecting a single option from a fixed action set, conditional decision-making in compositional action space requires generating an action under explicit conditions.

Figure 2: The overview of the dataset construction pipeline and the evaluation protocol of Condesion-Bench .

Model	DSR	CSR
Non-reasoning Models
GPT-4.1	0.5139	0.7968
GPT-4.1-mini	0.5299	0.7968
Claude-3.5-Sonnet	0.6375	0.8499
Claude-3.5-Haiku	0.5737	0.8234
Gemini-2.0-Flash	0.3227	0.6467

Table 1: Comparison of Decision Satisfaction Rate (DSR) and Condition Satisfaction Rate (CSR) across various LLMs. Bold indicates the best performance among each type of models.

Models	V	C	A
Non-reasoning Models
GPT-4.1-mini	91.63	76.10	71.31
GPT-4.1	89.64	79.68	69.72
Claude-3.5-Haiku	90.04	85.66	71.31
Claude-3.5-Sonnet	90.44	93.63	70.92
Gemini-2.0-Flash	90.84	77.29	63.35

Table 2: Satisfaction rate (%) for each condition type. V, C, A refer to variable, contextual, and allocation conditions, respectively. Bold indicates the best performance among each type of models.

Figure 3: Decision Satisfaction Rate (DSR) over all generated actions and Normalized Utility (NU) computed on condition-satisfying actions.

Figure 4: Comparison of performance regarding the number of generated action candidates. The upper plot indicates the proportion of cases in which at least one action satisfies all conditions, while the lower plot reports NU of the best action among the satisfying ones.

Figure 5: Decision Satisfaction Rate (DSR) over all generated actions and Normalized ROI (NR) computed on condition-violating actions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	SNU	FNU	DSR	CSR
Non-reasoning Models
Nova-Pro	0.4215	0.4797	0.3680	0.7173
Nova-Flash	0.4509	0.4621	0.2640	0.6347
Reasoning Models
Claude-4.5-Sonnet	0.5048	0.4963	0.5968	0.8199
Claude-4.5-Haiku	0.4492	0.4468	0.5680	0.8427

Appendix

Table 3: Additional results on Condesion-Bench conducted with zero-shot prompting. SNU denotes NU of successful actions that satisfy conditions and FNU denotes NU of failed actions.

Condition Type	Natural Language Form
1. Variable Conditions
Individual	I should buy stocks including {stock} .
	I should buy stocks except for {stock} .
Number	I should buy more than {num} kinds of stocks.
	I should buy less than {num} kinds of stocks.
Sector	I should only buy the stock of the company regarding {sector} .

Appendix

Table 4: Types and manually generated natural language formats of conditions

\mathcal{C}

Figure 6: Sectors and following stock (decision variable) lists used for Condesion-Bench .

Figure 7: Mapping of stock tickers to company names used for Condesion-Bench construction.

Figure 8: Qualitative example of scenario $\mathcal{S}$ , condition $\mathcal{C}$ , action $a$ (feasible action) and evaluation on Condesion-Bench .

Figure 9: Qualitative example of scenario $\mathcal{S}$ , condition $\mathcal{C}$ , action $a$ (infeasible action) and evaluation on Condesion-Bench .

Reasoning	Type	Model	Source	Knowledge cutoff	Size
X	Proprietary	gpt-4.1-2025-04-14	( OpenAI, 2024 )	2024.06	-
		gpt-4.1-mini-2025-04-14	( OpenAI, 2024 )	2024.06	-
		Claude-3.5-Sonnet-2024-10-22	( Anthropic, 2024a )	2024.04	-
		Claude-3.5-Haiku-2024-10-22	( Anthropic, 2024b )	2024.07	-
		gemini-2.0-flash-001	( Gemini Team, 2023 )	2024.06	-
		gemini-2.0-flash-lite-001	( Gemini Team, 2023 )	2024.06	-

Appendix

Table 5: Details on the models used for our experiments. For models that do not explicitly disclose a knowledge cutoff, the release date is used as a proxy and is marked with *.

Figure 10: Prompt for summarizing overall market situation.

Figure 11: Prompt for generating final action. Input data are given in Figure 8 and 9 .

Figure 12: Prompt for sampling action candidates. Input data are given in Figure 8 and 9 and the number of samples is shown in Figure 4 .

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

When Do We Need LLMs? A Diagnostic for Language-Driven Bandits
Apr 7, 2026Uljad Berdica, Fernando Acero, Anton Ipsen +3Contextual Bandits LLM Decision-Making
University of Oxford · J.P. Morgan AI Research
JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent Harnesses LLM Agents
University of Illinois Urbana-Champaign · Princeton University · Google
Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety Benchmarks LLM Evaluation
1Independent Researcher · University of Sheffield

Apr 7, 2026J/K move · Enter open · S save

When Do We Need LLMs? A Diagnostic for Language-Driven Bandits

Uljad Berdica, Fernando Acero, Anton Ipsen, Parisa Zehtabi+2

University of Oxford · J.P. Morgan AI Research

Contextual Bandits LLM Decision-Making LLM Uncertainty Estimation

Open paper AskPDF arXiv

We study Contextual Multi-Armed Bandits (CMABs) for non-episodic decision-making problems where the context includes both textual and numerical information (e.g., recommendation systems, dynamic portfolio adjustments, offer selection; all frequent problems in finance). While Large Language Models (LLMs) are increasingly applied to these settings, utilizing LLMs for reasoning at every decision step is computationally expensive, and uncertainty estimates are difficult to obtain. To address this, we introduce LLMP-UCB, a bandit algorithm that derives uncertainty estimates from LLMs via repeated inference. However, our experiments demonstrate that lightweight numerical bandits operating on text embeddings (dense or Matryoshka) match or exceed the accuracy of LLM-based solutions at a fraction of their cost. We further show that embedding dimensionality is a practical lever on the exploration-exploitation balance, enabling cost-performance tradeoffs without prompt complexity. Finally, to guide practitioners, we propose a geometric diagnostic based on the arms' embeddings to decide when to use LLM-driven reasoning versus a lightweight numerical bandit. Our results provide a principled deployment framework for cost-effective, uncertainty-aware decision systems with broad applicability across AI use cases.

Abstract

LLM Decision-Making

Explore similar work

When Do We Need LLMs? A Diagnostic for Language-Driven Bandits

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation