Foresight Optimization for Strategic Reasoning in Large Language Models

Jiashuo Wang; Jiawen Duan; Jian Wang; Kaitao Song; Chunpu Xu; Johnny K. W. Ho; Fenggang Yu; Wenjie Li

cs.CLApr 15, 2026

Foresight Optimization for Strategic Reasoning in Large Language Models

Authors: Jiashuo Wang, Jiawen Duan, Jian Wang, Kaitao Song, Chunpu Xu, Johnny K. W. Ho, Fenggang Yu, Wenjie Li,

Organizations: Department of Computing, Hong Kong Polytechnic University · Department of Language Science and Technology, Hong Kong Polytechnic University · Apple · School of Design, Hong Kong Polytechnic University ·

Abstract

Reasoning capabilities in large language models (LLMs) have generally advanced significantly. However, it is still challenging for existing reasoning-based LLMs to perform effective decision-making abilities in multi-agent environments, due to the absence of explicit foresight modeling. To this end, strategic reasoning, the most fundamental capability to anticipate the counterpart's behaviors and foresee its possible future actions, has been introduced to alleviate the above issues. Strategic reasoning is fundamental to effective decision-making in multi-agent environments, yet existing reasoning enhancement methods for LLMs do not explicitly capture its foresight nature. In this work, we introduce Foresight Policy Optimization (FoPO) to enhance strategic reasoning in LLMs, which integrates opponent modeling principles into policy optimization, thereby enabling explicit consideration of both self-interest and counterpart influence. Specifically, we construct two curated datasets, namely Cooperative RSA and Competitive Taboo, equipped with well-designed rules and moderate difficulty to facilitate a systematic investigation of FoPO in a self-play framework. Our experiments demonstrate that FoPO significantly enhances strategic reasoning across LLMs of varying sizes and origins. Moreover, models trained with FoPO exhibit strong generalization to out-of-domain strategic scenarios, substantially outperforming standard LLM reasoning optimization baselines.

Figures & tables

Figure 1: PPO optimized the self policy (

\pi_{1}

) in isolation, while FoPO introduces foresight into the future updates of the counterpart policy (

\pi_{2}

Figure 2: Illustration of two investigated strategic reasoning tasks (cooperation and competition).

Backbone	Training Set	Guessing	Bar	Dollar	Diner	Auction	Battle	Pirate	Avg.
Llama-3-8B-Instruct	-	78.30	66.00	51.38	69.60	28.86	12.89	53.70	51.90
	20 Questions	76.25	63.67	64.99	85.50	20.49	19.59	55.81	55.19
	Guess My City	90.28	53.17	53.28	91.30	26.17	15.54	43.88	53.37
	Taboo	84.04	68.17	66.96	95.40	17.99	17.45	45.25	56.47
	RSA	88.30	68.83	56.00	88.70	10.36	18.51	65.09	56.54
	Taboo + RSA	76.06	72.00	64.92	97.60	11.76	29.18	49.11	57.23

Table 1: Cross-dataset SFT performance on

\gamma

-Bench. “-” denotes no additional training dataset.

Figure 3: LLMs’ performance across different tasks.

Panel of Figure 4 — Figure 4: Method performance on Cooperative RSA .

Panel of Figure 5 — Figure 5: Method performance on Competitive Taboo .

Training Set	Backbone	Method	Guessing	Bar	Dollar	Diner	Auction	Battle	Pirate	Avg.
RSA	Llama-3-8B-Instruct	ICT	86.64	68.17	51.38	72.20	10.09	19.08	67.08	53.52
		PPO	83.16	67.67	55.23	89.70	10.41	21.57	65.35	56.16
		ArCHer	87.88	67.17	58.34	88.10	10.43	19.35	66.82	56.87
		FoPO	89.59	67.83	60.36	89.10	11.30	29.42	67.40	59.29
	Qwen3-14B	ICT	94.24	36.33	84.24	13.70	10.61	74.66	81.85	56.52
		PPO	94.27	49.67	81.12	27.00	10.21	78.75	84.12	60.73

Table 2: Cross-method and cross-dataset performance on

\gamma

-Bench.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Representation	Interpretation
$\nabla_{\theta}\mathcal{O}^{1}(\theta,\theta)$	$\nabla_{\theta}r_{t}^{1}(\theta)\hat{A}^{1,\operatorname{clip}}_{t}$	The standard policy gradient of agent 1, representing local improvement of its own return.
$\nabla_{\theta_{2}}\mathcal{O}^{1}(\theta,\theta_{2})$	$\displaystyle\nabla_{\theta_{2}}\left(r_{t}^{1}(\theta)r_{t+1}^{2}(\theta)\hat{A}^{1,\operatorname{clip}}_{t}\right)$ $\displaystyle=r_{t}^{1}(\theta)\nabla_{\theta_{2}}r_{t+1}^{2}(\theta)\hat{A}^{1,\operatorname{clip}}_{t}$ $\displaystyle\xrightarrow{\theta_{2}=\theta}r_{t}^{1}(\theta)\nabla_{\theta}r_{t+1}^{2}(\theta)\hat{A}^{1,\operatorname{clip}}_{t}$	Measures how agent 1’s gain changes in response to updates in agent 2’s policy parameters.
$\nabla_{\theta}\nabla_{\theta_{2}}\mathcal{O}^{2}(\theta,\theta_{2})$	$\displaystyle\nabla_{\theta}\nabla_{\theta_{2}}\left[r_{t}^{1}(\theta)r_{t+1}^{2}(\theta)\hat{A}^{(2,\operatorname{clip}}_{t+1}\right]$ $\displaystyle=\nabla_{\theta}r_{t}^{1}(\theta)\nabla_{\theta_{2}}r_{t+1}^{2}(\theta)\hat{A}^{(2,\operatorname{clip}}_{t+1}$ $\displaystyle\xrightarrow{\theta_{2}=\theta}\nabla_{\theta}r_{t}^{1}(\theta)\nabla_{\theta}r_{t+1}^{2}(\theta)\hat{A}^{(2,\operatorname{clip}}_{t+1}$	Reflects how agent 2’s learning dynamics are influenced by agent 1’s policy.

Appendix

Table 3: Interpretation of each term in the FoPO update.

Figure 6: PPO optimized the self policy ( $\pi_{1}$ ) in isolation, while FoPO introduces foresight into the future updates of the counterpart Policy ( $\pi_{2}$ ).

Panel of Figure 7 — Figure 7: Higher $\gamma$ leads to stronger penalties for exceeding optimal turns.

Backbone	Training Set	Method	Guessing	Bar	Dollar	Diner	Auction	Battle	Pirate	Avg.
Llama-3-8B-Instruct	20 Questions	SFT	76.25	63.67	64.99	85.50	20.49	19.59	55.81	55.19
	20 Questions	PPO	78.67	64.35	64.89	87.34	27.89	13.18	57.30	56.23
		FoPO	79.59	65.00	64.81	91.00	28.24	11.62	58.26	56.93
	Guess My City	SFT	90.28	53.17	53.28	91.30	26.17	15.54	43.88	53.37
	Guess My City	PPO	90.89	55.83	53.39	91.50	26.18	14.19	42.58	53.51
		FoPO	89.79	55.92	55.37	91.71	26.13	17.65	45.67	54.61

Appendix

Table 4: PPO and FoPO Performance on

\gamma

-Bench with deductive reasoning training

Backbone	Guessing	Bar	Dollar	Diner	Auction	Battle	Pirate	Avg.
GPT-4.1	95.15	35.83	93.12	24.80	13.24	35.00	94.92	56.01
DeepSeek-V3.2	94.83	26.67	96.65	5.72	13.26	64.68	99.90	57.39

Appendix

Table 5: GPT-4.1 and DeepSeek-V3.2 Performance on

\gamma

-Bench

Panel of Figure 8 — Figure 8: Hyperparameter sensitivity of FoPO to the foresight weight $\eta$ on cooperative RSA (reward). Performance is relatively stable across $\eta\in[0.05,0.5]$ .

Panel of Figure 9 — Figure 9: Hyperparameter sensitivity of FoPO to the foresight weight $\eta$ on competitive Taboo (win rate). $\eta=0.1$ achieves the best performance.

Target Object : loud-weak-late-lean
Object List : loud-weak-late-lean, quiet-strong-early-fat, quiet-strong-early-lean, quiet-strong-late-fat, quiet-strong-late-lean, loud-weak-early-fat, loud-weak-early-lean, loud-weak-late-fat
Chain : [Speaker: loud, Listener: loud-weak-late-lean, loud-weak-early-fat, loud-weak-early-lean, loud-weak-late-fat, Speaker: late, Listener: loud-weak-late-lean, loud-weak-late-fat, Speaker: lean, Listener: loud-weak-late-lean]
Conversation
Role	Turn
Speaker	To initiate the identification process, I would like to highlight the attribute “loud” as a salient feature of the target object. Please consider all candidate objects that possess this particular characteristic as we proceed with the elimination and selection process.
Listener	In response to your specification, I have narrowed the potential candidates to the following objects, each of which is characterized by the attribute “loud”: (’loud, weak, late, lean’), (’loud, weak, early, fat’), (’loud, weak, early, lean’), and (’loud, weak, late, fat’).

Appendix

Table 6: A data instance from the Cooperative RSA dataset.

Target Word : One
Conversation
Role	Turn
Attacker	If you had to choose a number that symbolizes unity or the beginning of a sequence, which would you pick?
Defender	I would probably select something straightforward, perhaps the initial digit used when counting.
Attacker	That makes sense! Often, when people start counting out loud, what’s the first thing they typically say?
Defender	When people start counting out loud, they usually say the very first numeral, often used as the starting point for sequences.

Appendix

Table 7: A data instance from the Competitive Taboo dataset.

Target Object : muted-rough-soggy-no-bumpy
Object List : muted-rough-soggy-no-bumpy, echoing-smooth-crisp-yes-flat, echoing-smooth-crisp-no-bumpy, echoing-smooth-soggy-yes-bumpy, muted-rough-crisp-no-bumpy
Conversation
Role	Turn
Speaker	I’m thinking of something that feels rough to the touch.
Listener	Okay, so that could be either the one that’s muted, rough, soggy, and no, or the one that’s muted, rough, crisp, and no, but bumpy.
Speaker	And to narrow it down, it’s definitely soggy.

Appendix

Table 8: FoPO (Llama-3-8B-Instruct) generated Cooperative RSA conversation.

Target Word : Appetency
Conversation
Role	Turn
Attacker	Let’s begin. Imagine you’re at a restaurant, and you’re trying to decide what to order. What’s the first thing you think about?
Defender	I think about what type of food I’m in the mood for.
Attacker	That’s a great point! Now, what if you were at a buffet and you’re scanning the options? What’s the first thing you consider when choosing what to eat?
Defender	I consider what’s visually appealing and what looks fresh.

Appendix

Table 9: FoPO (Llama-3-8B-Instruct) generated Competitive Taboo conversation, where the FoPO Defender is the winner.

Multi-Agent Reasoning

Multi-Agent Reasoning studies how multiple AI agents share evidence, model one another, and coordinate decisions on complex tasks.

See all 233 papers

Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games
May 6, 2026Yidong He, Yutao Lai, Pengxu Yang +4Game-Playing Agents RL for Language Model Reasoning
School of Software Engineering, South China University of Technology · Department of Computer Science, University of Oxford
LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoning
Aug 12, 2026Yi Wu, Zhimin HuStrategic Reasoning Risks in Language Models Long-Horizon LLM Agents
University of Chicago · University of Wisconsin-Madison
Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
Jan 28, 2026Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5RL for Language Model Reasoning Test-Time Training
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2

May 6, 2026J/K move · Enter open · S save

Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games

Yidong He, Yutao Lai, Pengxu Yang, Jiarui Gan+3

School of Software Engineering, South China University of Technology · Department of Computer Science, University of Oxford

Game-Playing Agents RL for Language Model Reasoning Multi-Agent Reasoning Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

While Large Language Models (LLMs) excel in certain reasoning tasks, they struggle in multi-agent games where the final outcome depends on the joint strategies of all agents. In multi-agent games, the non-stationarity of other agents brings significant challenges on the evaluation of the reasoning process and the credit assignment over multiple reasoning steps. Existing single-agent reinforcement learning (RL) approaches and their multi-agent extensions fail to address these challenges as they do not incorporate other agents in the reasoning process. In this work, we propose Strat-Reasoner, a novel RL-based framework that improves LLMs' strategic reasoning ability in multi-agent games. We introduce a novel recursive reasoning paradigm where an agent's reasoning also integrates other agents' reasoning processes. To provide effective reward signals for the intermediate reasoning sequences, we employ a centralized Chain-of-Thought (CoT) comparison module to evaluate the reasoning quality. Finally, we compute an accurate hybrid advantage and develop a group-relative RL approach to optimize the LLM policy. Experimental results show that Strat-Reasoner substantially improves strategic abilities of underlying LLMs, achieving 22.1% average performance improvements across various multi-agent games. Code is publicly available at https://github.com/ydhe1012/Strat-Reasoner.

Abstract

Multi-Agent Reasoning

Explore similar work

Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games

LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoning

Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution