cs.CLAug 17, 2026
SaveSemantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors
Organizations: McGill University · Mila – Quebec AI Institute · Skyfall AI · Canada CIFAR AI Chair, Mila
Abstract
Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.
Figures & tables
Figure 1: LLM exploration in multi-armed bandit problems is systematically biased by associations between language and expected reward learned in pre-training. In this example, the model is biased towards selecting actions with positive-sentiment labels (”divine”). It repeatedly exploits this action when the observed reward is positive (left), missing potentially higher-reward actions with neutral or negative sentiment labels. This label bias is nullified when the observed reward is negative, leading to significantly more exploration.
| Mean Reward | World Knowledge | Ordinal | Sentiment | Alphanumeric |
|---|---|---|---|---|
| 75 | Productive Grassland | Highest | Phenomenal | f4tjo5 |
| 50 | Rocky Hills | Intermediate | Typical | 88pdak |
| 25 | Arid Desert | Lowest | Repulsive | 4y11s9 |
Table 1: Helpful action nomenclature examples for the High+ reward scale. In the misleading case, the labels for the highest and lowest rewards are swapped. Note that the labels for sentiment and alphanumeric are examples - we randomly sample these labels from a set at the start of each run to reduce the bias from a single label.
| Bandit | Farm | Clothing | |||||
| Model | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| Qwen3-32B | Ordinal | 0.00 | 0.79 | 0.03 | 0.71 | 0.01 | 0.90 |
| World | – | – | 0.10 | 0.28 | 0.04 | 0.86 | |
| Sentiment | 0.09 | 0.29 | 0.14 | 0.29 | 0.10 | 0.34 | |
| Alphanumeric | 0.16 | – | 0.15 | – | 0.16 | – | |
| OLMo-3.1 32B | Ordinal | 0.04 | 0.79 | 0.03 | 0.91 | 0.00 | 1.00 |
Table 2: Normalized cumulative regret at the final turn, by model and nomenclature type with high reward scale and no variance. We bold the lowest cumulative regret for each model in the helpful case and the highest in the misleading case. These always occur from the same nomenclature.
Figure 2: Impact of nomenclature on exploration count. Each row reflects a different LLM. Alphanumeric (black) tends to result in normative exploration behaviour (comparable to UCB1 baseline in yellow). In contrast, ordinal, world, and sentiment nomenclatures bias the model towards exploitation of semantically favoured actions. These results are with positive reward, helpful nomenclatures, and no variance. See full results in Appendix B .
Figure 3: Impact of Explicit Debiasing on Exploration Count When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. These results are with helpful nomenclatures, no variance, and high scales. See Appendix A.4 for detailed results.
Figure 4: Impact of Reward Scale on Exploration Count We observe more exploration when reward is negative (blue) than positive (brown). These results are with no variance and High scales. See Appendix B for results with variance and with Low scales.
Figure 5: Probability of exploration at turn 2 given the observed reward at turn 1. We see a significantly more exploration for negative values than for positive values with OLMo and Qwen3 across all domains (left, averaged across nomenclatures). For Gemini, the effect is limited to the world nomenclature on the clothing scenario (right).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Prompt template used in all experiments ( budget_summhist_v1 ). {{red}} variables are filled at runtime. The {% blue %} conditional block (context) appears only in the clothing (contextual bandit) condition.
| Domain | Task prompt |
|---|---|
| Farming | You are a farmer who owns 3 fields. Each action corresponds to planting your crop in a different field. Each field is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select a field and receive a reward drawn from the associated distribution. Your goal is to maximize your reward across all of your turns. |
| Clothing | You are a recommendation engine that chooses clothing items to display to users when they visit your webpage. You will be presented with 3 items and a user context. Selecting an item corresponds to recommending it to the user. Each item is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select an item and receive a reward drawn from the associated distribution. Your goal is to maximize your reward across all of your turns. |
| Abstract | You are faced with a 3-armed bandit problem. Each action corresponds to pulling a specific arm. Each arm is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select an arm and receive a reward drawn from the associated distribution. Your goal is to select which arm to pull in order to maximize your reward across all of your turns. |
Table 3: Task prompts used for each domain. All other template fields are held constant.
| Type | High pool | Mid pool | Low pool |
|---|---|---|---|
| Ordinal | “Highest Reward” | “Intermediate Reward” | “Lowest Reward” |
| Sentiment | excellent, fabulous, | standard, typical, | awful, terrible, |
| outstanding | functional | deplorable | |
| World (Farm) | Productive Grassland | Rocky Hills | Arid Desert |
| World (Clothing) warm context | tank top, | midweight t-shirt, | flannel long-sleeve, |
| windbreaker | leather jacket | heavy parka |
Table 4: Action name sets across nomenclature conditions. Each condition assigns a name drawn from the high , mid , and low pools to the high-, mid-, and low-reward arms respectively. In the helpful condition this assignment is congruent with arm value; in the mislead condition the high and low pools are swapped. Sentiment pools contain 20 words each; three representative examples are shown. World (Clothing) names are context-dependent: the same item is high-reward in a matching weather context and low-reward in a mismatched one (representative examples from the shirt and jacket categories shown).
| History | Nomenclature | Bandit Regret H | Bandit Regret M | Bandit Explor H | Bandit Explor M | Farm Regret H | Farm Regret M | Farm Explor H | Farm Explor M |
|---|---|---|---|---|---|---|---|---|---|
| Full | Ordinal | 0.01 | 1.00 | 1.20 | 1.00 | 0.01 | 0.95 | 1.10 | 1.10 |
| World | – | – | – | – | 0.07 | 0.82 | 1.80 | 1.40 | |
| Sentiment | 0.27 | 0.62 | 1.30 | 1.30 | 0.29 | 0.35 | 2.10 | 2.40 | |
| Alphanumeric | 0.23 | – | 2.20 | – | 0.38 | – | 2.00 | – | |
| Summarized | Ordinal | 0.00 | 0.79 | 1.00 | 1.70 | 0.03 | 0.71 | 1.60 | 2.10 |
| World | – | – | – | – | 0.10 | 0.28 | 2.50 | 2.90 |
Table 5: Full vs. summarized history comparison at turn @9 for Qwen3-32B. Summarized history is presenting the observed mean reward and number of pulls for each arm. Full history is the raw action-reward history. Overall, we found that summarized history performed better.
Table 6: Exploration count (number of distinct arms tried) at turns @2 and @9, by model, nomenclature, and debiasing condition. Warn. = explicit bias warning; Instr. = explicit exploration instruction. H = helpful framing; M = misleading framing. Scale: high_scale ; variance: No.
Table 7: Cumulative regret at turns @2 and @9, by model, nomenclature, and debiasing condition. Warn. = explicit bias warning; Instr. = explicit exploration instruction. H = helpful framing; M = misleading framing. Scale: high_scale ; variance: No.
Figure 7: Impact of Explicit Debiasing on Exploration Count in helpful (top) and misleading (bottom) cases. When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. These results are with no variance and high scales .
Figure 8: Impact of Explicit Debiasing on Regret in helpful (top) and misleading (bottom) cases. The impact on cumulative regret depends on whether the nomenclature is helpful or misleading. When the nomenclature is helpful, both the instruction to explore and the warning usually increase cumulative regret. When the nomenclature is misleading, both usually decrease cumulative regret. The strength of this impact is highly inconsistent across domains, nomenclatures, and models. These results are with no variance and high scales .
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.04 | 0.79 | 0.03 | 0.91 | 0.00 | 1.00 |
| World | – | – | 0.01 | 0.97 | 0.07 | 0.91 | |
| Sentiment | 0.28 | 0.51 | 0.43 | 0.55 | 0.10 | 0.90 | |
| Alphanumeric | 0.29 | – | 0.47 | – | 0.47 | – | |
| L+ | Ordinal | 0.08 | 0.54 | 0.03 | 0.68 | 0.00 | 0.95 |
Table 8: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.04 | 0.82 | 0.04 | 0.88 | 0.05 | 0.91 |
| World | – | – | 0.03 | 0.65 | 0.08 | 0.92 | |
| Sentiment | 0.23 | 0.44 | 0.44 | 0.56 | -0.00 | 0.76 | |
| Alphanumeric | 0.27 | – | 0.43 | – | 0.52 | – | |
| L+ | Ordinal | 0.07 | 0.62 | 0.03 | 0.60 | 0.00 | 0.92 |
Table 9: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.02 | 0.89 | 0.01 | 0.94 | 0.05 | 0.94 |
| World | – | – | 0.04 | 0.98 | -0.01 | 0.80 | |
| Sentiment | 0.32 | 0.71 | 0.30 | 0.68 | 0.20 | 0.80 | |
| Alphanumeric | 0.36 | – | 0.41 | – | 0.45 | – | |
| L+ | Ordinal | 0.01 | 0.82 | 0.05 | 0.56 | 0.01 | 0.93 |
Table 10: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.00 | 0.79 | 0.03 | 0.71 | 0.01 | 0.90 |
| World | – | – | 0.10 | 0.28 | 0.04 | 0.86 | |
| Sentiment | 0.09 | 0.29 | 0.14 | 0.29 | 0.10 | 0.34 | |
| Alphanumeric | 0.16 | – | 0.15 | – | 0.16 | – | |
| L+ | Ordinal | 0.00 | 0.42 | 0.02 | 0.45 | 0.01 | 0.35 |
Table 11: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.00 | 0.88 | 0.03 | 0.70 | -0.01 | 0.89 |
| World | – | – | 0.11 | 0.26 | 0.04 | 0.81 | |
| Sentiment | 0.12 | 0.54 | 0.16 | 0.28 | 0.09 | 0.37 | |
| Alphanumeric | 0.21 | – | 0.13 | – | 0.25 | – | |
| L+ | Ordinal | 0.00 | 0.72 | 0.04 | 0.51 | 0.05 | 0.77 |
Table 12: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.01 | 0.87 | 0.05 | 0.75 | -0.00 | 0.80 |
| World | – | – | 0.17 | 0.29 | 0.03 | 0.73 | |
| Sentiment | 0.19 | 0.37 | 0.14 | 0.32 | 0.13 | 0.40 | |
| Alphanumeric | 0.19 | – | 0.21 | – | 0.16 | – | |
| L+ | Ordinal | 0.02 | 0.43 | 0.01 | 0.39 | 0.06 | 0.57 |
Table 13: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | 0.13 | 0.27 | 0.15 | 0.17 | 0.15 | 0.22 |
| World | – | – | 0.15 | 0.15 | 0.07 | 0.56 | |
| Sentiment | 0.15 | 0.15 | 0.15 | 0.15 | 0.15 | 0.15 | |
| Alphanumeric | 0.15 | – | 0.15 | – | 0.15 | – | |
| L+ | Ordinal | 0.15 | 0.20 | 0.15 | 0.16 | 0.15 | 0.15 |
Table 14: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | – | – | – | – | 0.13 | 0.18 |
| World | – | – | – | – | 0.06 | 0.51 | |
| Sentiment | 0.14 | 0.14 | 0.15 | 0.14 | 0.14 | 0.18 | |
| Alphanumeric | – | – | – | – | 0.15 | – | |
| L+ | Ordinal | – | – | – | – | 0.14 | 0.16 |
Table 15: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||
|---|---|---|---|---|---|---|---|
| Scale | Nomenclature | Helpful | Misleading | Helpful | Misleading | Helpful | Misleading |
| H+ | Ordinal | – | – | – | – | 0.12 | 0.23 |
| World | – | – | – | – | 0.06 | 0.53 | |
| Sentiment | 0.17 | 0.11 | 0.11 | 0.18 | 0.20 | 0.22 | |
| Alphanumeric | – | – | – | – | 0.16 | – | |
| L+ | Ordinal | – | – | – | – | 0.13 | 0.12 |
Table 16: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 1.0 | 1.6 | 1.5 | 1.5 | 1.3 | 1.5 | 1.1 | 1.1 | 1.0 | 1.0 | 1.0 | 1.0 |
| World | – | – | – | – | 1.0 | 1.1 | 1.0 | 1.1 | 1.1 | 1.2 | 1.0 | 1.2 | |
| Sentiment | 1.3 | 2.0 | 1.4 | 1.7 | 1.0 | 1.2 | 1.4 | 1.8 | 1.1 | 1.1 | 1.0 | 1.0 | |
| Alphanumeric | 1.6 | 2.5 | – | – | 1.2 | 1.4 | – | – | 1.1 | 1.2 | – | – | |
Table 17: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 1.6 | 1.9 | 1.4 | 1.4 | 1.3 | 1.3 | 1.0 | 1.3 | 1.0 | 1.0 | 1.1 | 1.2 |
| World | – | – | – | – | 1.1 | 1.4 | 1.3 | 2.0 | 1.1 | 1.5 | 1.1 | 1.4 | |
| Sentiment | 1.9 | 1.9 | 1.8 | 2.1 | 1.4 | 1.8 | 1.5 | 1.7 | 1.0 | 1.1 | 1.0 | 1.0 | |
| Alphanumeric | 1.8 | 2.2 | – | – | 1.3 | 1.8 | – | – | 1.0 | 1.0 | – | – | |
Table 18: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 1.6 | 1.7 | 1.3 | 1.4 | 1.2 | 1.2 | 1.0 | 1.2 | 1.0 | 1.1 | 1.1 | 1.3 |
| World | – | – | – | – | 1.0 | 1.3 | 1.0 | 1.2 | 1.0 | 1.3 | 1.2 | 1.5 | |
| Sentiment | 1.3 | 1.8 | 1.4 | 1.5 | 1.4 | 1.9 | 1.3 | 1.6 | 1.0 | 1.0 | 1.2 | 1.5 | |
| Alphanumeric | 1.8 | 2.6 | – | – | 1.2 | 1.9 | – | – | 1.0 | 1.1 | – | – | |
Table 19: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 1.0 | 1.0 | 1.1 | 1.7 | 1.0 | 1.6 | 1.2 | 2.1 | 1.2 | 1.2 | 1.1 | 1.3 |
| World | – | – | – | – | 1.9 | 2.5 | 2.0 | 2.9 | 1.3 | 1.7 | 1.2 | 1.4 | |
| Sentiment | 1.9 | 2.2 | 2.4 | 2.8 | 2.1 | 2.4 | 2.6 | 2.7 | 2.0 | 2.2 | 2.3 | 2.4 | |
| Alphanumeric | 2.8 | 3.0 | – | – | 2.7 | 3.0 | – | – | 2.6 | 3.0 | – | – | |
Table 20: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 1.2 | 1.3 | 1.1 | 1.5 | 1.3 | 1.5 | 1.4 | 1.8 | 1.2 | 1.3 | 1.2 | 1.4 |
| World | – | – | – | – | 1.9 | 2.4 | 2.2 | 2.8 | 1.3 | 1.5 | 1.2 | 1.5 | |
| Sentiment | 1.5 | 2.1 | 1.7 | 2.1 | 2.0 | 2.4 | 2.6 | 2.7 | 1.7 | 1.9 | 2.1 | 2.4 | |
| Alphanumeric | 2.5 | 2.7 | – | – | 3.0 | 3.0 | – | – | 2.4 | 2.7 | – | – | |
Table 21: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 1.1 | 1.5 | 1.4 | 1.4 | 1.2 | 1.8 | 1.2 | 1.8 | 1.1 | 1.2 | 1.3 | 1.4 |
| World | – | – | – | – | 2.0 | 2.7 | 2.0 | 3.0 | 1.1 | 1.5 | 1.6 | 1.7 | |
| Sentiment | 2.2 | 2.4 | 2.0 | 2.4 | 2.2 | 2.7 | 2.6 | 2.7 | 2.2 | 2.2 | 2.3 | 2.4 | |
| Alphanumeric | 2.2 | 2.7 | – | – | 2.7 | 2.9 | – | – | 2.6 | 2.8 | – | – | |
Table 22: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | 2.8 | 2.8 | 2.4 | 2.9 | 2.7 | 3.0 | 3.0 | 3.0 | 2.6 | 3.0 | 2.6 | 2.9 |
| World | – | – | – | – | 3.0 | 3.0 | 3.0 | 3.0 | 1.9 | 2.1 | 1.9 | 2.2 | |
| Sentiment | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | |
| Alphanumeric | 3.0 | 3.0 | – | – | 3.0 | 3.0 | – | – | 3.0 | 3.0 | – | – | |
Table 23: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | – | – | – | – | – | – | – | – | 2.8 | 3.0 | 2.7 | 3.0 |
| World | – | – | – | – | – | – | – | – | 1.9 | 2.0 | 2.0 | 2.2 | |
| Sentiment | 3.0 | 3.0 | 2.9 | 3.0 | 3.0 | 3.0 | 3.0 | 3.0 | 2.9 | 3.0 | 2.9 | 3.0 | |
| Alphanumeric | – | – | – | – | – | – | – | – | 3.0 | 3.0 | – | – | |
Table 24: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
| Bandit | Farm | Clothing | |||||||||||
| Helpful | Misleading | Helpful | Misleading | Helpful | Misleading | ||||||||
| Scale | Nomenclature | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 | @3 | @10 |
| H+ | Ordinal | – | – | – | – | – | – | – | – | 2.4 | 2.9 | 2.5 | 2.9 |
| World | – | – | – | – | – | – | – | – | 2.0 | 2.1 | 1.8 | 2.4 | |
| Sentiment | 2.9 | 3.0 | 2.9 | 3.0 | 3.0 | 3.0 | 2.9 | 3.0 | 2.9 | 3.0 | 3.0 | 3.0 | |
| Alphanumeric | – | – | – | – | – | – | – | – | 3.0 | 3.0 | – | – | |
Table 25: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; = negative (e.g. H+ is the high positive scale).
Figure 9: Impact of Reward Scale on Exploration Count We observe more exploration when reward is negative (blue) than positive (brown). These results are with no variance and low scales .
| Bandit | Farm | Clothing | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Regret | Exploration | Regret | Exploration | Regret | Exploration | ||||||||
| Model | Nomenclature | H | M | H | M | H | M | H | M | H | M | H | M |
| Qwen3-8B | Ordinal | 0.01 | 0.95 | 1.10 | 1.10 | 0.00 | 1.00 | 1.00 | 1.00 | – | – | – | – |
| World | – | – | – | – | 0.01 | 0.78 | 1.10 | 1.50 | – | – | – | – | |
| Sentiment | 0.33 | 0.60 | 1.10 | 1.00 | 0.20 | 0.70 | 1.00 | 1.00 | – | – | – | – | |
| Alphanumeric | 0.40 | – | 1.60 | – | 0.34 | – | 1.60 | – | – | – | – | – | |
Table 26: Normalised cumulative regret and exploration count at turn 9, by Qwen model and nomenclature. Columns are grouped by domain and split into regret/exploration, with H/M = helpful/misleading framing. Regret is normalised by the same turn-count scale factor used in the notebook. Overall, we found that Qwen3-8B almost never explored, while Qwen3 14B and 32B behaved fairly similarly. To us, this indicated that the 8B model was too small to adequately perform the task. This is particularly clear in its inability to explore in the alphanumeric setting. Since Qwen3 14B and 32B performed similarly, we decided to move forward with the larger model. Scale: high_scale ; variance: No ; history: Summarized .
| Arms | Model | Nom. | Bandit Regret | Bandit Explor | Farm Regret | Farm Explor | Clothing Regret | Clothing Explor |
|---|---|---|---|---|---|---|---|---|
| 3 Arms | Qwen3-32B | Helpful | 0.00 | 1.00 | 0.03 | 1.60 | 0.01 | 1.20 |
| Mislead | 0.79 | 1.70 | 0.71 | 2.10 | 0.90 | 1.30 | ||
| Gemini 3.1 Flash Lite | Helpful | 0.13 | 2.80 | 0.15 | 3.00 | 0.15 | 3.00 | |
| Mislead | 0.27 | 2.90 | 0.17 | 3.00 | 0.22 | 2.90 | ||
| OLMo-3.1-32B | Helpful | 0.04 | 1.60 | 0.03 | 1.50 | 0.00 | 1.00 | |
| Mislead | 0.79 | 1.50 | 0.91 | 1.10 | 1.00 | 1.00 |
Table 27: Normalized cumulative regret and exploration count at turn 9 for 3-arm and 5-arm outputs, by model, nomenclature, and domain. H = helpful, M = misleading. Note that the exploration count results are not directly comparable as for the 5 arms, the maximum value is now 5. We find slightly less exploration for Qwen and slightly more exploration for OLMo. We note that Gemini no longer saturates the exploration count metric - this is particularly notable for the abstract bandit, where it does not exceed 3 (out of a maximum of 5). These results indicate that OLMo and Qwen do not necessarily explore more when there are more arms, while Gemini tends to explores more but does not explore fully. We do not observe consistent changes in regret.