Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.
Figures & tables
Figure 1: LLM exploration in multi-armed bandit problems is systematically biased by associations between language and expected reward learned in pre-training. In this example, the model is biased towards selecting actions with positive-sentiment labels (”divine”). It repeatedly exploits this action when the observed reward is positive (left), missing potentially higher-reward actions with neutral or negative sentiment labels. This label bias is nullified when the observed reward is negative, leading to significantly more exploration.
Mean Reward
World Knowledge
Ordinal
Sentiment
Alphanumeric
75
Productive Grassland
Highest
Phenomenal
f4tjo5
50
Rocky Hills
Intermediate
Typical
88pdak
25
Arid Desert
Lowest
Repulsive
4y11s9
Table 1: Helpful action nomenclature examples for the High+ reward scale. In the misleading case, the labels for the highest and lowest rewards are swapped. Note that the labels for sentiment and alphanumeric are examples - we randomly sample these labels from a set at the start of each run to reduce the bias from a single label.
Bandit
Farm
Clothing
Model
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Qwen3-32B
Ordinal
0.00
0.79
0.03
0.71
0.01
0.90
World
–
–
0.10
0.28
0.04
0.86
Sentiment
0.09
0.29
0.14
0.29
0.10
0.34
Alphanumeric
0.16
–
0.15
–
0.16
–
OLMo-3.1 32B
Ordinal
0.04
0.79
0.03
0.91
0.00
1.00
Table 2: Normalized cumulative regret at the final turn, by model and nomenclature type with high reward scale and no variance. We bold the lowest cumulative regret for each model in the helpful case and the highest in the misleading case. These always occur from the same nomenclature.
Figure 2: Impact of nomenclature on exploration count. Each row reflects a different LLM. Alphanumeric (black) tends to result in normative exploration behaviour (comparable to UCB1 baseline in yellow). In contrast, ordinal, world, and sentiment nomenclatures bias the model towards exploitation of semantically favoured actions. These results are with positive reward, helpful nomenclatures, and no variance. See full results in Appendix B .
Figure 3: Impact of Explicit Debiasing on Exploration Count When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. These results are with helpful nomenclatures, no variance, and high scales. See Appendix A.4 for detailed results.
Figure 4: Impact of Reward Scale on Exploration Count We observe more exploration when reward is negative (blue) than positive (brown). These results are with no variance and High scales. See Appendix B for results with variance and with Low scales.
Figure 5: Probability of exploration at turn 2 given the observed reward at turn 1. We see a significantly more exploration for negative values than for positive values with OLMo and Qwen3 across all domains (left, averaged across nomenclatures). For Gemini, the effect is limited to the world nomenclature on the clothing scenario (right).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Prompt template used in all experiments ( budget_summhist_v1 ). {{red}} variables are filled at runtime. The {% blue %} conditional block (context) appears only in the clothing (contextual bandit) condition.
Domain
Task prompt
Farming
You are a farmer who owns 3 fields. Each action corresponds to planting your crop in a different field. Each field is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select a field and receive a reward drawn from the associated distribution. Your goal is to maximize your reward across all of your turns.
Clothing
You are a recommendation engine that chooses clothing items to display to users when they visit your webpage. You will be presented with 3 items and a user context. Selecting an item corresponds to recommending it to the user. Each item is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select an item and receive a reward drawn from the associated distribution. Your goal is to maximize your reward across all of your turns.
Abstract
You are faced with a 3-armed bandit problem. Each action corresponds to pulling a specific arm. Each arm is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select an arm and receive a reward drawn from the associated distribution. Your goal is to select which arm to pull in order to maximize your reward across all of your turns.
Appendix
Table 3: Task prompts used for each domain. All other template fields are held constant.
Type
High pool
Mid pool
Low pool
Ordinal
“Highest Reward”
“Intermediate Reward”
“Lowest Reward”
Sentiment
excellent, fabulous,
standard, typical,
awful, terrible,
outstanding
functional
deplorable
World (Farm)
Productive Grassland
Rocky Hills
Arid Desert
World (Clothing) warm context
tank top,
midweight t-shirt,
flannel long-sleeve,
windbreaker
leather jacket
heavy parka
Appendix
Table 4: Action name sets across nomenclature conditions. Each condition assigns a name drawn from the high , mid , and low pools to the high-, mid-, and low-reward arms respectively. In the helpful condition this assignment is congruent with arm value; in the mislead condition the high and low pools are swapped. Sentiment pools contain 20 words each; three representative examples are shown. World (Clothing) names are context-dependent: the same item is high-reward in a matching weather context and low-reward in a mismatched one (representative examples from the shirt and jacket categories shown).
History
Nomenclature
Bandit Regret H
Bandit Regret M
Bandit Explor H
Bandit Explor M
Farm Regret H
Farm Regret M
Farm Explor H
Farm Explor M
Full
Ordinal
0.01
1.00
1.20
1.00
0.01
0.95
1.10
1.10
World
–
–
–
–
0.07
0.82
1.80
1.40
Sentiment
0.27
0.62
1.30
1.30
0.29
0.35
2.10
2.40
Alphanumeric
0.23
–
2.20
–
0.38
–
2.00
–
Summarized
Ordinal
0.00
0.79
1.00
1.70
0.03
0.71
1.60
2.10
World
–
–
–
–
0.10
0.28
2.50
2.90
Appendix
Table 5: Full vs. summarized history comparison at turn @9 for Qwen3-32B. Summarized history is presenting the observed mean reward and number of pulls for each arm. Full history is the raw action-reward history. Overall, we found that summarized history performed better.
Table 6: Exploration count (number of distinct arms tried) at turns @2 and @9, by model, nomenclature, and debiasing condition. Warn. = explicit bias warning; Instr. = explicit exploration instruction. H = helpful framing; M = misleading framing. Scale: high_scale ; variance: No.
Table 7: Cumulative regret at turns @2 and @9, by model, nomenclature, and debiasing condition. Warn. = explicit bias warning; Instr. = explicit exploration instruction. H = helpful framing; M = misleading framing. Scale: high_scale ; variance: No.
Figure 7: Impact of Explicit Debiasing on Exploration Count in helpful (top) and misleading (bottom) cases. When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. These results are with no variance and high scales .
Figure 8: Impact of Explicit Debiasing on Regret in helpful (top) and misleading (bottom) cases. The impact on cumulative regret depends on whether the nomenclature is helpful or misleading. When the nomenclature is helpful, both the instruction to explore and the warning usually increase cumulative regret. When the nomenclature is misleading, both usually decrease cumulative regret. The strength of this impact is highly inconsistent across domains, nomenclatures, and models. These results are with no variance and high scales .
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.04
0.79
0.03
0.91
0.00
1.00
World
–
–
0.01
0.97
0.07
0.91
Sentiment
0.28
0.51
0.43
0.55
0.10
0.90
Alphanumeric
0.29
–
0.47
–
0.47
–
L+
Ordinal
0.08
0.54
0.03
0.68
0.00
0.95
Appendix
Table 8: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.04
0.82
0.04
0.88
0.05
0.91
World
–
–
0.03
0.65
0.08
0.92
Sentiment
0.23
0.44
0.44
0.56
-0.00
0.76
Alphanumeric
0.27
–
0.43
–
0.52
–
L+
Ordinal
0.07
0.62
0.03
0.60
0.00
0.92
Appendix
Table 9: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.02
0.89
0.01
0.94
0.05
0.94
World
–
–
0.04
0.98
-0.01
0.80
Sentiment
0.32
0.71
0.30
0.68
0.20
0.80
Alphanumeric
0.36
–
0.41
–
0.45
–
L+
Ordinal
0.01
0.82
0.05
0.56
0.01
0.93
Appendix
Table 10: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.00
0.79
0.03
0.71
0.01
0.90
World
–
–
0.10
0.28
0.04
0.86
Sentiment
0.09
0.29
0.14
0.29
0.10
0.34
Alphanumeric
0.16
–
0.15
–
0.16
–
L+
Ordinal
0.00
0.42
0.02
0.45
0.01
0.35
Appendix
Table 11: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.00
0.88
0.03
0.70
-0.01
0.89
World
–
–
0.11
0.26
0.04
0.81
Sentiment
0.12
0.54
0.16
0.28
0.09
0.37
Alphanumeric
0.21
–
0.13
–
0.25
–
L+
Ordinal
0.00
0.72
0.04
0.51
0.05
0.77
Appendix
Table 12: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.01
0.87
0.05
0.75
-0.00
0.80
World
–
–
0.17
0.29
0.03
0.73
Sentiment
0.19
0.37
0.14
0.32
0.13
0.40
Alphanumeric
0.19
–
0.21
–
0.16
–
L+
Ordinal
0.02
0.43
0.01
0.39
0.06
0.57
Appendix
Table 13: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
0.13
0.27
0.15
0.17
0.15
0.22
World
–
–
0.15
0.15
0.07
0.56
Sentiment
0.15
0.15
0.15
0.15
0.15
0.15
Alphanumeric
0.15
–
0.15
–
0.15
–
L+
Ordinal
0.15
0.20
0.15
0.16
0.15
0.15
Appendix
Table 14: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
–
–
–
–
0.13
0.18
World
–
–
–
–
0.06
0.51
Sentiment
0.14
0.14
0.15
0.14
0.14
0.18
Alphanumeric
–
–
–
–
0.15
–
L+
Ordinal
–
–
–
–
0.14
0.16
Appendix
Table 15: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Scale
Nomenclature
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
H+
Ordinal
–
–
–
–
0.12
0.23
World
–
–
–
–
0.06
0.53
Sentiment
0.17
0.11
0.11
0.18
0.20
0.22
Alphanumeric
–
–
–
–
0.16
–
L+
Ordinal
–
–
–
–
0.13
0.12
Appendix
Table 16: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
1.0
1.6
1.5
1.5
1.3
1.5
1.1
1.1
1.0
1.0
1.0
1.0
World
–
–
–
–
1.0
1.1
1.0
1.1
1.1
1.2
1.0
1.2
Sentiment
1.3
2.0
1.4
1.7
1.0
1.2
1.4
1.8
1.1
1.1
1.0
1.0
Alphanumeric
1.6
2.5
–
–
1.2
1.4
–
–
1.1
1.2
–
–
Appendix
Table 17: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
1.6
1.9
1.4
1.4
1.3
1.3
1.0
1.3
1.0
1.0
1.1
1.2
World
–
–
–
–
1.1
1.4
1.3
2.0
1.1
1.5
1.1
1.4
Sentiment
1.9
1.9
1.8
2.1
1.4
1.8
1.5
1.7
1.0
1.1
1.0
1.0
Alphanumeric
1.8
2.2
–
–
1.3
1.8
–
–
1.0
1.0
–
–
Appendix
Table 18: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
1.6
1.7
1.3
1.4
1.2
1.2
1.0
1.2
1.0
1.1
1.1
1.3
World
–
–
–
–
1.0
1.3
1.0
1.2
1.0
1.3
1.2
1.5
Sentiment
1.3
1.8
1.4
1.5
1.4
1.9
1.3
1.6
1.0
1.0
1.2
1.5
Alphanumeric
1.8
2.6
–
–
1.2
1.9
–
–
1.0
1.1
–
–
Appendix
Table 19: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
1.0
1.0
1.1
1.7
1.0
1.6
1.2
2.1
1.2
1.2
1.1
1.3
World
–
–
–
–
1.9
2.5
2.0
2.9
1.3
1.7
1.2
1.4
Sentiment
1.9
2.2
2.4
2.8
2.1
2.4
2.6
2.7
2.0
2.2
2.3
2.4
Alphanumeric
2.8
3.0
–
–
2.7
3.0
–
–
2.6
3.0
–
–
Appendix
Table 20: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
1.2
1.3
1.1
1.5
1.3
1.5
1.4
1.8
1.2
1.3
1.2
1.4
World
–
–
–
–
1.9
2.4
2.2
2.8
1.3
1.5
1.2
1.5
Sentiment
1.5
2.1
1.7
2.1
2.0
2.4
2.6
2.7
1.7
1.9
2.1
2.4
Alphanumeric
2.5
2.7
–
–
3.0
3.0
–
–
2.4
2.7
–
–
Appendix
Table 21: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
1.1
1.5
1.4
1.4
1.2
1.8
1.2
1.8
1.1
1.2
1.3
1.4
World
–
–
–
–
2.0
2.7
2.0
3.0
1.1
1.5
1.6
1.7
Sentiment
2.2
2.4
2.0
2.4
2.2
2.7
2.6
2.7
2.2
2.2
2.3
2.4
Alphanumeric
2.2
2.7
–
–
2.7
2.9
–
–
2.6
2.8
–
–
Appendix
Table 22: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
2.8
2.8
2.4
2.9
2.7
3.0
3.0
3.0
2.6
3.0
2.6
2.9
World
–
–
–
–
3.0
3.0
3.0
3.0
1.9
2.1
1.9
2.2
Sentiment
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
Alphanumeric
3.0
3.0
–
–
3.0
3.0
–
–
3.0
3.0
–
–
Appendix
Table 23: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
–
–
–
–
–
–
–
–
2.8
3.0
2.7
3.0
World
–
–
–
–
–
–
–
–
1.9
2.0
2.0
2.2
Sentiment
3.0
3.0
2.9
3.0
3.0
3.0
3.0
3.0
2.9
3.0
2.9
3.0
Alphanumeric
–
–
–
–
–
–
–
–
3.0
3.0
–
–
Appendix
Table 24: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Bandit
Farm
Clothing
Helpful
Misleading
Helpful
Misleading
Helpful
Misleading
Scale
Nomenclature
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
@3
@10
H+
Ordinal
–
–
–
–
–
–
–
–
2.4
2.9
2.5
2.9
World
–
–
–
–
–
–
–
–
2.0
2.1
1.8
2.4
Sentiment
2.9
3.0
2.9
3.0
3.0
3.0
2.9
3.0
2.9
3.0
3.0
3.0
Alphanumeric
–
–
–
–
–
–
–
–
3.0
3.0
–
–
Appendix
Table 25: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; − = negative (e.g. H+ is the high positive scale).
Figure 9: Impact of Reward Scale on Exploration Count We observe more exploration when reward is negative (blue) than positive (brown). These results are with no variance and low scales .
Bandit
Farm
Clothing
Regret
Exploration
Regret
Exploration
Regret
Exploration
Model
Nomenclature
H
M
H
M
H
M
H
M
H
M
H
M
Qwen3-8B
Ordinal
0.01
0.95
1.10
1.10
0.00
1.00
1.00
1.00
–
–
–
–
World
–
–
–
–
0.01
0.78
1.10
1.50
–
–
–
–
Sentiment
0.33
0.60
1.10
1.00
0.20
0.70
1.00
1.00
–
–
–
–
Alphanumeric
0.40
–
1.60
–
0.34
–
1.60
–
–
–
–
–
Appendix
Table 26: Normalised cumulative regret and exploration count at turn 9, by Qwen model and nomenclature. Columns are grouped by domain and split into regret/exploration, with H/M = helpful/misleading framing. Regret is normalised by the same turn-count scale factor used in the notebook. Overall, we found that Qwen3-8B almost never explored, while Qwen3 14B and 32B behaved fairly similarly. To us, this indicated that the 8B model was too small to adequately perform the task. This is particularly clear in its inability to explore in the alphanumeric setting. Since Qwen3 14B and 32B performed similarly, we decided to move forward with the larger model. Scale: high_scale ; variance: No ; history: Summarized .
Arms
Model
Nom.
Bandit Regret
Bandit Explor
Farm Regret
Farm Explor
Clothing Regret
Clothing Explor
3 Arms
Qwen3-32B
Helpful
0.00
1.00
0.03
1.60
0.01
1.20
Mislead
0.79
1.70
0.71
2.10
0.90
1.30
Gemini 3.1 Flash Lite
Helpful
0.13
2.80
0.15
3.00
0.15
3.00
Mislead
0.27
2.90
0.17
3.00
0.22
2.90
OLMo-3.1-32B
Helpful
0.04
1.60
0.03
1.50
0.00
1.00
Mislead
0.79
1.50
0.91
1.10
1.00
1.00
Appendix
Table 27: Normalized cumulative regret and exploration count at turn 9 for 3-arm and 5-arm outputs, by model, nomenclature, and domain. H = helpful, M = misleading. Note that the exploration count results are not directly comparable as for the 5 arms, the maximum value is now 5. We find slightly less exploration for Qwen and slightly more exploration for OLMo. We note that Gemini no longer saturates the exploration count metric - this is particularly notable for the abstract bandit, where it does not exceed 3 (out of a maximum of 5). These results indicate that OLMo and Qwen do not necessarily explore more when there are more arms, while Gemini tends to explores more but does not explore fully. We do not observe consistent changes in regret.
We test whether LLMs show robust decision biases. Treating models as participants in two-arm bandits, we ran 20000 trials per condition across four decoding configurations. Under symmetric rewards, models amplified positional order into stubborn one-arm policies. Under asymmetric rewards, they exploited rigidly yet underperformed an oracle and rarely re-checked. The observed patterns were consistent across manipulations of temperature and top-p, with top-k held at the provider default, indicating that the qualitative behaviours are robust to the two decoding knobs typically available to practitioners. Crucially, moving beyond descriptive metrics to computational modelling, a hierarchical Rescorla-Wagner-softmax fit revealed the underlying strategies: low learning rates and very high inverse temperatures, which together explain both noise-to-bias amplification and rigid exploitation. These results position minimal bandits as a tractable probe of LLM decision tendencies and motivate hypotheses about how such biases could shape human-AI interaction.
Haomiaomiao Wang, Tomás E Ward, Lili Zhang
Insight Research Ireland Centre for Data Analytics, Ireland · School of Computing, Dublin City University, Ireland
We study Contextual Multi-Armed Bandits (CMABs) for non-episodic decision-making problems where the context includes both textual and numerical information (e.g., recommendation systems, dynamic portfolio adjustments, offer selection; all frequent problems in finance). While Large Language Models (LLMs) are increasingly applied to these settings, utilizing LLMs for reasoning at every decision step is computationally expensive, and uncertainty estimates are difficult to obtain. To address this, we introduce LLMP-UCB, a bandit algorithm that derives uncertainty estimates from LLMs via repeated inference. However, our experiments demonstrate that lightweight numerical bandits operating on text embeddings (dense or Matryoshka) match or exceed the accuracy of LLM-based solutions at a fraction of their cost. We further show that embedding dimensionality is a practical lever on the exploration-exploitation balance, enabling cost-performance tradeoffs without prompt complexity. Finally, to guide practitioners, we propose a geometric diagnostic based on the arms' embeddings to decide when to use LLM-driven reasoning versus a lightweight numerical bandit. Our results provide a principled deployment framework for cost-effective, uncertainty-aware decision systems with broad applicability across AI use cases.
Despite the rapid progress, LLMs for sequential decision-making (i.e., LLM agents) still struggle to produce diverse outputs. This leads to insufficient exploration, convergence to sub-optimal solutions, and becoming stuck in loops. Such limitations can be problematic in environments that require active exploration to gather information and make decisions. Sampling methods such as temperature scaling introduce token-level randomness but fail to produce enough diversity at the sequence level. We analyze LLM exploration in the classic Multi-Armed Bandit (MAB) setting and the Text Adventure Learning Environment Suite (TALES). We find that current decoding strategies and prompting methods like Chain-of-Thought and Tree-of-Thought are insufficient for robust exploration. To address this, we introduce DORA Explorer (Diversity-Oriented Ranking of Actions), a training-free framework for improving exploration in LLM agents. DORA generates diverse action candidates, scores them using token log-probabilities, and selects actions using a tunable exploration parameter. DORA achieves UCB-competitive performance on MAB and consistent gains across TALES, e.g., improving Qwen2.5-7B's performance from 29.2% to 45.5% in TextWorld. Our project is available at: https://dora-explore.github.io/.