Foresight Optimization for Strategic Reasoning in Large Language Models
Authors: Jiashuo Wang, Jiawen Duan, Jian Wang, Kaitao Song, Chunpu Xu, Johnny K. W. Ho, Fenggang Yu, Wenjie Li, Johan F. Hoorn
Organizations: Department of Computing, Hong Kong Polytechnic University · Department of Language Science and Technology, Hong Kong Polytechnic University · Apple · School of Design, Hong Kong Polytechnic University · Research Institute for Quantum Technology, Hong Kong Polytechnic University · Department of Communication Science, Vrije Universiteit Amsterdam
Reasoning capabilities in large language models (LLMs) have generally advanced significantly. However, it is still challenging for existing reasoning-based LLMs to perform effective decision-making abilities in multi-agent environments, due to the absence of explicit foresight modeling. To this end, strategic reasoning, the most fundamental capability to anticipate the counterpart's behaviors and foresee its possible future actions, has been introduced to alleviate the above issues. Strategic reasoning is fundamental to effective decision-making in multi-agent environments, yet existing reasoning enhancement methods for LLMs do not explicitly capture its foresight nature. In this work, we introduce Foresight Policy Optimization (FoPO) to enhance strategic reasoning in LLMs, which integrates opponent modeling principles into policy optimization, thereby enabling explicit consideration of both self-interest and counterpart influence. Specifically, we construct two curated datasets, namely Cooperative RSA and Competitive Taboo, equipped with well-designed rules and moderate difficulty to facilitate a systematic investigation of FoPO in a self-play framework. Our experiments demonstrate that FoPO significantly enhances strategic reasoning across LLMs of varying sizes and origins. Moreover, models trained with FoPO exhibit strong generalization to out-of-domain strategic scenarios, substantially outperforming standard LLM reasoning optimization baselines.
Figures & tables
Figure 1: PPO optimized the self policy ( π1 ) in isolation, while FoPO introduces foresight into the future updates of the counterpart policy ( π2 ).
Figure 2: Illustration of two investigated strategic reasoning tasks (cooperation and competition).
Backbone
Training Set
Guessing
Bar
Dollar
Diner
Auction
Battle
Pirate
Avg.
Llama-3-8B-Instruct
-
78.30
66.00
51.38
69.60
28.86
12.89
53.70
51.90
20 Questions
76.25
63.67
64.99
85.50
20.49
19.59
55.81
55.19
Guess My City
90.28
53.17
53.28
91.30
26.17
15.54
43.88
53.37
Taboo
84.04
68.17
66.96
95.40
17.99
17.45
45.25
56.47
RSA
88.30
68.83
56.00
88.70
10.36
18.51
65.09
56.54
Taboo + RSA
76.06
72.00
64.92
97.60
11.76
29.18
49.11
57.23
Table 1: Cross-dataset SFT performance on γ -Bench. “-” denotes no additional training dataset.
Figure 3: LLMs’ performance across different tasks.
Figure 4: Method performance on Cooperative RSA .
Figure 5: Method performance on Competitive Taboo .
Training Set
Backbone
Method
Guessing
Bar
Dollar
Diner
Auction
Battle
Pirate
Avg.
RSA
Llama-3-8B-Instruct
ICT
86.64
68.17
51.38
72.20
10.09
19.08
67.08
53.52
PPO
83.16
67.67
55.23
89.70
10.41
21.57
65.35
56.16
ArCHer
87.88
67.17
58.34
88.10
10.43
19.35
66.82
56.87
FoPO
89.59
67.83
60.36
89.10
11.30
29.42
67.40
59.29
Qwen3-14B
ICT
94.24
36.33
84.24
13.70
10.61
74.66
81.85
56.52
PPO
94.27
49.67
81.12
27.00
10.21
78.75
84.12
60.73
Table 2: Cross-method and cross-dataset performance on γ -Bench.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Term
Representation
Interpretation
∇θO1(θ,θ)
∇θrt1(θ)A^t1,clip
The standard policy gradient of agent 1, representing local improvement of its own return.
Reflects how agent 2’s learning dynamics are influenced by agent 1’s policy.
Appendix
Table 3: Interpretation of each term in the FoPO update.
Figure 6: PPO optimized the self policy ( π1 ) in isolation, while FoPO introduces foresight into the future updates of the counterpart Policy ( π2 ).
Figure 7: Higher γ leads to stronger penalties for exceeding optimal turns.
Backbone
Training Set
Method
Guessing
Bar
Dollar
Diner
Auction
Battle
Pirate
Avg.
Llama-3-8B-Instruct
20 Questions
SFT
76.25
63.67
64.99
85.50
20.49
19.59
55.81
55.19
PPO
78.67
64.35
64.89
87.34
27.89
13.18
57.30
56.23
FoPO
79.59
65.00
64.81
91.00
28.24
11.62
58.26
56.93
Guess My City
SFT
90.28
53.17
53.28
91.30
26.17
15.54
43.88
53.37
PPO
90.89
55.83
53.39
91.50
26.18
14.19
42.58
53.51
FoPO
89.79
55.92
55.37
91.71
26.13
17.65
45.67
54.61
Appendix
Table 4: PPO and FoPO Performance on γ -Bench with deductive reasoning training
Backbone
Guessing
Bar
Dollar
Diner
Auction
Battle
Pirate
Avg.
GPT-4.1
95.15
35.83
93.12
24.80
13.24
35.00
94.92
56.01
DeepSeek-V3.2
94.83
26.67
96.65
5.72
13.26
64.68
99.90
57.39
Appendix
Table 5: GPT-4.1 and DeepSeek-V3.2 Performance on γ -Bench
Figure 8: Hyperparameter sensitivity of FoPO to the foresight weight η on cooperative RSA (reward). Performance is relatively stable across η∈[0.05,0.5] .
Figure 9: Hyperparameter sensitivity of FoPO to the foresight weight η on competitive Taboo (win rate). η=0.1 achieves the best performance.
To initiate the identification process, I would like to highlight the attribute “loud” as a salient feature of the target object. Please consider all candidate objects that possess this particular characteristic as we proceed with the elimination and selection process.
Listener
In response to your specification, I have narrowed the potential candidates to the following objects, each of which is characterized by the attribute “loud”: (’loud, weak, late, lean’), (’loud, weak, early, fat’), (’loud, weak, early, lean’), and (’loud, weak, late, fat’).
Appendix
Table 6: A data instance from the Cooperative RSA dataset.
Target Word : One
Conversation
Role
Turn
Attacker
If you had to choose a number that symbolizes unity or the beginning of a sequence, which would you pick?
Defender
I would probably select something straightforward, perhaps the initial digit used when counting.
Attacker
That makes sense! Often, when people start counting out loud, what’s the first thing they typically say?
Defender
When people start counting out loud, they usually say the very first numeral, often used as the starting point for sequences.
Appendix
Table 7: A data instance from the Competitive Taboo dataset.
Target Object : muted-rough-soggy-no-bumpy
Object List : muted-rough-soggy-no-bumpy, echoing-smooth-crisp-yes-flat, echoing-smooth-crisp-no-bumpy, echoing-smooth-soggy-yes-bumpy, muted-rough-crisp-no-bumpy
Conversation
Role
Turn
Speaker
I’m thinking of something that feels rough to the touch.
Listener
Okay, so that could be either the one that’s muted, rough, soggy, and no, or the one that’s muted, rough, crisp, and no, but bumpy.
Let’s begin. Imagine you’re at a restaurant, and you’re trying to decide what to order. What’s the first thing you think about?
Defender
I think about what type of food I’m in the mood for.
Attacker
That’s a great point! Now, what if you were at a buffet and you’re scanning the options? What’s the first thing you consider when choosing what to eat?
Defender
I consider what’s visually appealing and what looks fresh.
Appendix
Table 9: FoPO (Llama-3-8B-Instruct) generated Competitive Taboo conversation, where the FoPO Defender is the winner.
While Large Language Models (LLMs) excel in certain reasoning tasks, they struggle in multi-agent games where the final outcome depends on the joint strategies of all agents. In multi-agent games, the non-stationarity of other agents brings significant challenges on the evaluation of the reasoning process and the credit assignment over multiple reasoning steps. Existing single-agent reinforcement learning (RL) approaches and their multi-agent extensions fail to address these challenges as they do not incorporate other agents in the reasoning process. In this work, we propose Strat-Reasoner, a novel RL-based framework that improves LLMs' strategic reasoning ability in multi-agent games. We introduce a novel recursive reasoning paradigm where an agent's reasoning also integrates other agents' reasoning processes. To provide effective reward signals for the intermediate reasoning sequences, we employ a centralized Chain-of-Thought (CoT) comparison module to evaluate the reasoning quality. Finally, we compute an accurate hybrid advantage and develop a group-relative RL approach to optimize the LLM policy. Experimental results show that Strat-Reasoner substantially improves strategic abilities of underlying LLMs, achieving 22.1% average performance improvements across various multi-agent games. Code is publicly available at https://github.com/ydhe1012/Strat-Reasoner.
Yidong He, Yutao Lai, Pengxu Yang +4
School of Software Engineering, South China University of Technology · Department of Computer Science, University of Oxford
Strategic reasoning in Large Language Models (LLMs) within long-horizon environments is often limited by inconsistent subgoals. In these settings, finite attention resources prevent the model from maintaining strategic coherence over thousands of steps. This limitation leads to strategic drift, where localized decisions fail to sustain a coherent trajectory across reasoning. To address this, we introduce EpicStar, a framework that enables agents to learn memory as policy to tackle long-horizon reasoning. Specifically, the agent maintains a bank of successful past episodes as a heuristic alongside a working memory to track short-term environmental changes. During inference, a dynamic gating mechanism determines whether to execute a retrieved action directly or to perform new reasoning through a contextual fusion of the retrieved episodes and current working memory. Utilizing StarCraft II as the testbed, we evaluated EpicStar against diverse opponent styles. It significantly outperforms baseline methods, achieving higher win rates while consuming an order of magnitude fewer tokens, and it maintains this advantage consistently across difficulty levels and opponent strategies. Our findings provide compelling evidence that structured cross-episode memory is essential for enabling LLM agents to perform robust, long-term strategic execution in dynamic, autonomous settings.
Yi Wu, Zhimin Hu
University of Chicago · University of Wisconsin-Madison
Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.
Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2