Non-stationary environments require agents to revise previously learned action values when contingencies change. We treat large language models (LLMs) as sequential decision policies in a two-option probabilistic reversal-learning task with three latent states and switch events triggered by either a performance criterion or timeout. We compare a deterministic fixed transition cycle to a stochastic random schedule that increases volatility, and evaluate DeepSeek-V3.2, Gemini-3, and GPT-5.2, with human data as a behavioural reference. Across models, win-stay was near ceiling while lose-shift was markedly attenuated, revealing asymmetric use of positive versus negative evidence. DeepSeek-V3.2 showed extreme perseveration after reversals and weak acquisition, whereas Gemini-3 and GPT-5.2 adapted more rapidly but still remained less loss-sensitive than humans. Random transitions amplified reversal-specific persistence across LLMs yet did not uniformly reduce total wins, demonstrating that high aggregate payoff can coexist with rigid adaptation. Hierarchical reinforcement-learning (RL) fits indicate dissociable mechanisms: rigidity can arise from weak loss learning, inflated policy determinism, or value polarisation via counterfactual suppression. These results motivate reversal-sensitive diagnostics and volatility-aware models for evaluating LLMs under non-stationary uncertainty.
Figures & tables
Provider
Model
API model ID
Temperature
top- p
DeepSeek
DeepSeek-V3.2
deepseek-chat
1.0
1.0
Google
Gemini-3-Flash
gemini-3-flash-preview
1.0
1.0
OpenAI
GPT-5.2
gpt-5.2
1.0
1.0
Table 1: Evaluated LLMs and decoding settings used for data generation.
Figure 1: Random schedule: post-reversal adoption of the target option and event-aligned choice proportions around reversals.
Agent
Schedule
Lose-shift
Perseveration
Post-rev. regret
Total wins
DeepSeek-V3.2
Fixed
0.035
19.63
811.73
84.06
DeepSeek-V3.2
Random
0.038
27.39
459.69
117.58
Gemini-3
Fixed
0.235
7.24
403.43
126.11
Gemini-3
Random
0.210
10.05
316.82
136.10
GPT-5.2
Fixed
0.241
7.34
459.27
119.98
GPT-5.2
Random
0.209
10.90
369.23
131.13
Table 2: Core behavioural measures by agent and schedule. Post-reversal regret is computed from known reward probabilities over a 20-trial window following each true state switch and is reported in expected coins (higher = worse).
Figure 2: Posterior group-level parameter densities under Dual RL and Dual RL- κ DU (left vs. right rows) for fixed and random schedules (bottom vs. top rows). Panels show gain and loss learning rates ( ηpos,ηneg ), inverse temperature ( β ), and (for κ DU) the counterfactual updating parameter ( κ ).
We test whether LLMs show robust decision biases. Treating models as participants in two-arm bandits, we ran 20000 trials per condition across four decoding configurations. Under symmetric rewards, models amplified positional order into stubborn one-arm policies. Under asymmetric rewards, they exploited rigidly yet underperformed an oracle and rarely re-checked. The observed patterns were consistent across manipulations of temperature and top-p, with top-k held at the provider default, indicating that the qualitative behaviours are robust to the two decoding knobs typically available to practitioners. Crucially, moving beyond descriptive metrics to computational modelling, a hierarchical Rescorla-Wagner-softmax fit revealed the underlying strategies: low learning rates and very high inverse temperatures, which together explain both noise-to-bias amplification and rigid exploitation. These results position minimal bandits as a tractable probe of LLM decision tendencies and motivate hypotheses about how such biases could shape human-AI interaction.
Haomiaomiao Wang, Tomás E Ward, Lili Zhang
Insight Research Ireland Centre for Data Analytics, Ireland · School of Computing, Dublin City University, Ireland
Adaptive test-time compute for LLM agents aims to invoke extra computation only when it improves performance. Existing methods typically use confidence-, uncertainty-, or difficulty-based gates, assuming a fixed direction from the gating signal through compute need to the value of computation. This makes gating a utility-calibration problem: gating signals should align with whether extra computation improves the final outcome over the base policy. We show that this alignment is unstable: the same signal predicts rollout benefit in one setting and rollout harm in another, with reversals across environments and backbones even when the task is fixed. Wrong-direction gates can therefore worsen performance by precisely selecting harmful states. This reversal reflects a deeper distinction between compute need and compute suitability: a high uncertainty signal may indicate decision-difficult states where rollouts help compare alternatives, or intervention-unsuitable states where the current context does not support useful rollout-based improvement. Under this two-source model, fixed-direction gates are unreliable across heterogeneous settings. To address this, we propose DIAL (Direction-Informed Adaptive Learning), a sparse gate trained from signal-agnostic counterfactual exploration to learn the utility direction of state features per (environment, backbone). Across six environments and three backbones, DIAL yields a stronger overall success-cost trade-off than fixed-direction baselines.
Ziming Li, Jiatan Huang, Xiaoguang Guo +2
University of Connecticut · New Jersey Institute of Technology
Closed-loop revision is increasingly used in large language model (LLM) applications, but failures may reflect incomplete feedback or ineffective responses to correct feedback. We introduce a fixed-budget revision protocol with deterministic verifiers that report all remaining violations across exact-length, lexical, and compositional constraints. Fixing feedback correctness and completeness isolates model-side revision behavior. Across 19 open- and closed-source models, controller-level mean final joint success ranges from 17.4% to 99.8%, with substantial cross-model gaps persisting under identical initial drafts. Controlled experiments reveal reproducible model-specific responses to exact feedback. Post-training and scale reshape these responses without consistently bringing them closer to exact correction. Across all constraint families, failed trajectories often repeat earlier outputs, and prior recurrence is associated with lower subsequent recoverability. Matched-state interventions show that removing earlier dialogue while holding the current draft and feedback fixed changes recurrence escape without reliably improving final success; effects depend on the model, task, and trigger-state composition. Exact feedback makes revision errors observable, but does not make the closed loop reliable. Code and reproduction instructions: https://github.com/kevinjiang0121-cyber/exact-feedback-code.
Haitong Jiang, Chunlin Liu, Yile Wang +1
College of Computer Science and Software Engineering, Shenzhen University