CoupVisor: Strategy Optimization by Round and Challenge Decision Support

Cris Huynh

cs.AIAug 16, 2026

CoupVisor: Strategy Optimization by Round and Challenge Decision Support

Authors: Cris Huynh

Abstract

This paper presents CoupVisor, a decision-support system for the hidden-information card game Coup. It addresses two questions: what a player should do on each turn, and when a player should challenge an opponent's claim. The system is built around a single description of game events, which is shared across manual play, replay of recorded games, simulation, belief tracking, advisor recommendations, and learning-based policies. CoupVisor estimates the chance that a claim is truthful by combining how likely each role is with how many cards the claimant still holds, which corrects a case where the very first claim of a game was flagged as suspicious despite no evidence. We compare a rule-following advisor and several learned and heuristic players across many simulated games and different opponent styles. Our main finding is that the choice of reward, whether it rewards short-term gains or ultimately winning the game, decides which learning approach performs best, and that a win-oriented reward produces a policy that outperforms all baselines.

Figures & tables

Fig. 1: The CoupVisor pipeline. Public events enter from any of three sources, manual turn entry, replay of a recorded game, or the synthetic simulator, and are applied by the engine, which maintains the public state: coins, remaining influence, revealed roles, the current claim, and the last public action. The belief tracker then updates the hidden-role distribution for every player, and the advisor combines public state and belief into a player-facing recommendation. Because all three input sources produce the same event schema, the advice path is identical in live play, in replay, and during policy training.

Fig. 2: Round-by-round strategy optimization loop. Each round begins by estimating hidden roles from the current belief state, then enumerates the actions that are legal given the player’s coins and influence, scores each candidate by expected utility against the risk that it is blocked or challenged, and returns a ranked suggestion. The legality filter runs before scoring, so constraints such as the forced Coup at ten coins remove options rather than merely penalising them.

Fig. 3: Challenge decision logic. On observing a claim the advisor reads $p_{\mathrm{truth}}$ from the belief model, checks how many copies of the claimed role remain unaccounted for, and applies the threshold rule. The two branches lead to Challenge and Do not challenge , each of which is turned into a human-readable advice sentence carrying the probability and the threshold that produced it, so the recommendation can be audited rather than merely obeyed.

Fig. 4: Hidden-role belief update pipeline. An incoming event multiplies the score of the role it implicates by the corresponding likelihood ratio, all scores are then damped by deck pressure and normalised to a distribution, and finally reveal constraints are applied. The two modes diverge only at the last step: in Standard mode duplicate roles in one hand are permitted, so a revealed role can still carry non-zero hidden probability, whereas in Strict no-duplicate-hand mode a player down to a single hidden influence has the revealed role forced to zero and the remainder renormalised.

Fig. 5: Single-turn sequence. The turn opens with an action and its implied role claim, after which the table may block, challenge the action, or challenge the block. Each branch produces its own events, and a challenge resolves before the action or counteraction it targets. Every arrow that carries a claim or a reveal triggers a belief update and a fresh advisor recommendation, which is why a single turn can generate several decision points rather than one.

Step	Player	$P(\mathrm{Duke})$	$P(\mathrm{Assassin})$	$P(\mathrm{Captain})$	$P(\mathrm{Ambassador})$	$P(\mathrm{Contessa})$
0	A	0.20	0.20	0.20	0.20	0.20
0	B, C, D	0.20	0.20	0.20	0.20	0.20
1	A	0.55	0.15	0.10	0.10	0.10
1	B, C, D	0.15	0.20	0.20	0.20	0.25
2	A	0.30	0.40	0.10	0.10	0.10
2	B	0.15	0.15	0.20	0.20	0.30

TABLE I: Belief evolution over the worked example. Step 0 is the uniform prior, Step 1 follows A’s Duke claim, Step 2 follows A’s Assassin claim.

Step	Event	Advisor update
1	A claims Duke (Tax)	Increase $P_{A}(\mathrm{Duke})$
2	No challenge	Belief remains uncertain
3	A claims Assassin	Reduce joint-claim consistency
4	Decision point	Compute $p_{\mathrm{truth}}$ , compare EV

TABLE II: Event sequence and corresponding advisor update.

Fig. 6: Training and evaluation pipeline. Simulated games are flattened into state, action, reward, next-state transitions, which feed both the behavior cloning head and the offline DQN. Both learned policies and the non-learning baselines are then replayed through the same evaluation harness on the same transitions, so that reward per episode, reward per step, and action-match rate are computed identically for every policy.

Fig. 7: User interaction flow. The user loads or enters a game, steps through it event by event, and at each step sees the reconstructed public state, the hidden-role belief table, and the advisor recommendation for any claim currently on the table. Advice and validation warnings are presented side by side, so that a probabilistic judgement is never confused with a hard rule violation.

Seed	Aggressive-heavy	Honest-heavy
101	50 games	50 games
202	50 games	50 games
303	50 games	50 games
404	50 games	50 games
505	50 games	50 games
Total	250 games	250 games

TABLE III: Primary balanced matrix (seed

\times

condition).

Metric	Mean	Std	CI low	CI high
Challenge F1	0.4236	0.1409	0.3363	0.5110
Outcome accuracy	0.5694	0.0242	0.5544	0.5844
Outcome score	0.1428	0.0486	0.1127	0.1730

TABLE IV: Advisor metrics across seed-condition cells.

Condition	Challenge F1	Outcome acc.	Outcome score
Aggressive-heavy	0.5641	0.5619	0.1239
Honest-heavy	0.2831	0.5769	0.1617

TABLE V: Condition comparison by bot mix (mean across seeds).

Condition	Weighted score
Aggressive-heavy	0.5628
Honest-heavy	0.4594

TABLE VI: Weighted ranking score by condition.

Mode	Challenge F1	Outcome acc.	Outcome score
Standard	0.4236	0.5694	0.1428
Strict no-duplicate	0.4190	0.5684	0.1408
$\Delta$ (Strict $-$ Std.)	$-0.0046$	$-0.0010$	$-0.0020$

TABLE VII: Standard vs strict no-duplicate-hand belief mode.

Mode	Challenge F1	Outcome acc.	Outcome score
Advisor on	0.4236	0.5694	0.1428
Advisor off	0.0000	0.8192	0.6424
$\Delta$ (Off $-$ On)	$-0.4236$	$+0.2498$	$+0.4996$

TABLE VIII: Advisor policy ablation (off = never-challenge baseline).

Policy	Reward/episode	Reward/step	Action match
DQN	$-0.2868$	$-0.0179$	0.1905
Random	$-0.6004$	$-0.0375$	0.1415
Behavior cloning	$1.0412$	$0.0651$	0.3980

TABLE IX: Policy benchmark under the original shaped reward (

\Delta

coins

+

\Delta

alive

+

action-match).

Policy	Reward/episode	Reward/step	Winner match
DQN (win-reward)	6.74	0.248	0.373
Belief-EV heuristic	3.68	0.135	0.263
Behavior cloning	3.38	0.124	0.330
Honest (baseline)	2.84	0.104	0.244
Rule-based heuristic	2.42	0.089	0.264
Pressure heuristic	2.30	0.085	0.245

TABLE X: Policy benchmark under win-aligned reward (

+1

when the chosen action matches the winner’s logged action). 100 games; 1000 training episodes for DQN.

Episodes	Avg ep. reward	Avg ep. steps	Final $\varepsilon$
10	12.700	22.60	0.95111
50	14.580	24.46	0.77213
100	14.840	24.46	0.60577
200	15.045	24.46	0.36695
500	16.278	24.46	0.08157
1000	17.793	24.46	0.00500

TABLE XI: DQN training summary under the win-aligned reward.

Fig. 8: Advisor metrics across the ten seed-condition cells. Each metric is plotted with its spread over cells, which makes visible the contrast reported in Table IV : outcome accuracy sits in a narrow band while challenge F1 varies widely, so the instability is confined to the challenge decision rather than affecting the advisor as a whole.

Fig. 9: Advisor performance split by opponent condition. The aggressive-heavy and honest-heavy pools are shown side by side for each metric. Challenge F1 separates sharply between the two while outcome accuracy barely moves, identifying opponent composition as the source of the F1 variance in Fig. 8 .

Fig. 10: DQN reward curve under the original shaped reward, 100 episodes. Per-episode reward oscillates between roughly $-6$ and $+4$ , and the rolling mean stays inside $[-3,+0.5]$ with no upward trend, so the policy is not learning. This flat curve is what motivated changing the reward rather than the algorithm. Corresponds to Table IX .

Fig. 11: DQN reward curve under the win-aligned reward, 1000 episodes. The rolling mean rises from about $1.5$ at episode 20 to about $5.9$ by episode 1000, a steady climb rather than the flat band of Fig. 10 , confirming that the same architecture does learn once the reward credits winner-matching actions. Corresponds to Table X .

Fig. 12: Policy benchmark summary. Every policy is scored on the same transitions, so the bars are directly comparable. Under the shaped reward the learned and heuristic policies cluster near or below zero while behavior cloning leads, the ordering that the win-aligned rerun in Table X subsequently inverts.

Fig. 13: Exploration-mode ablation under identical seeds, architecture, buffer, and reward. Left: running average episode reward. Middle: mean per-step entropy of the action distribution in nats. Right: action-visitation histogram. The $\varepsilon$ -greedy entropy curve is monotone in $\varepsilon$ and so is fixed entirely by the schedule, whereas the Boltzmann entropy curve also responds to the spread of learned action values and can plateau above the schedule floor when several actions have comparable value. The two mechanisms therefore allocate exploration differently even when their scalar schedules agree, which is why entropy rather than the schedule parameter is the quantity worth reporting.

Case	Evidence	System output	Ground truth and issue
E1	game 339, event 48 (Tax, Duke by P2)	Challenge ; $p_{\mathrm{truth}}=1.93\mathrm{e}{-34}$ , thr. $0.25$	Challenge result is a win for the claimant, so the claim was true. Overconfident false positive.
E2	game 231, event 1 (Steal block, Ambassador by P2)	Do not challenge ; $p_{\mathrm{truth}}=0.20$ , thr. $0.20$	Claimant loses, so the claim was false. Borderline equality missed a profitable challenge.
E3	game 158, event 7 (Steal, Captain by P3)	Do not challenge ; $p_{\mathrm{truth}}=0.2503$ , thr. $0.25$	Claimant loses, so the claim was false. Near-threshold miss.
E4	game 417, event 49 (Foreign Aid block, Duke by P4)	Challenge ; $p_{\mathrm{truth}}=8.98\mathrm{e}{-34}$ , thr. $0.35$	Claim was true. Block-claim handling is calibration-sensitive.
E5	DQN trace, game 106, step 12	DQN chose Exchange holding 11 coins	Legal action is Coup . In 10+ coin states DQN chose non-Coup 151 of 181 times, so forced-Coup alignment is weak.

TABLE XII: Representative failure cases from the evaluation run.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Network	2 hidden layers, width 128, ReLU
Optimiser	Adam, learning rate $1\times 10^{-3}$
Loss	Mean squared Bellman error
Discount factor $\gamma$	0.98
Batch size	64
Replay buffer size	10,000

Appendix

TABLE XIII: DQN training defaults.

Action	Effect	Can be blocked by
Income	Take 1 coin	Nobody
Foreign Aid	Take 2 coins	Duke
Coup	Pay 7 coins, target loses an influence	Nobody

Appendix

TABLE XIV: General actions, available to every player without a role claim.

Character	Action	Effect	Can be blocked by
Duke	Tax	Take 3 coins	Nobody
Assassin	Assassinate	Pay 3 coins, target loses an influence	Contessa
Captain	Steal	Take 2 coins from another player	Captain, Ambassador
Ambassador	Exchange	Draw 2 from the Court deck, return 2	Nobody

Appendix

TABLE XV: Character actions. Each requires claiming the listed character.

Character claimed	Blocks	Result
Duke	Foreign Aid	Target receives no coins that turn
Contessa	Assassination	Assassination fails, fee stays spent
Captain or Ambassador	Steal	Thief receives no coins that turn

Appendix

TABLE XVI: Counteractions.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 202 papers

Nov 10, 2025

Scalable Decision Making for Games of Imperfect Information

Real-world decision-making generally involves hidden information, that is, information that is unknown to one agent but possessed by another. Unfortunately, the presence of large amounts of hidden information renders established reinforcement learning and search approaches ineffective. Even with multimillion-dollar industrial research efforts, top-human-level play at Stratego---a board wargame with hidden information on a massive scale---has remained beyond the reach of artificial intelligence (AI). Here we introduce Ataraxos, an AI for Stratego based on general techniques that we developed for both self-play reinforcement learning and test-time search under hidden information. Ataraxos defeated the most decorated human Stratego player of all time by a large margin---achieving, to our knowledge, the first superhuman result in the game's history---while consuming orders of magnitude less compute and data than previous efforts. Using the same techniques, we built a superhuman AI for Barrage Stratego and state-of-the-art AIs for Hanabi and dou dizhu, all with low cost and high sample efficiency. The success of this approach across adversarial, cooperative and team games establishes a design pattern for reinforcement learning and search that is effective under large amounts of hidden information, a longstanding desideratum of the field of strategic decision-making.

Samuel Sokota, Eugene Vinitsky, Hengyuan Hu +3

Carnegie Mellon University · NYU Tandon School of Engineering · Stanford University +1

Game-Playing Agents Inference-Time Search Self-Play RL Imperfect-Information Games RL for Game AI

Ask PDF

Apr 28, 2026

StratFormer: Adaptive Opponent Modeling and Exploitation in Imperfect-Information Games

We present StratFormer, a transformer-based meta-agent that learns to simultaneously model and exploit opponents in imperfect-information games through a two-phase curriculum. The first phase trains an opponent modeling head to identify behavioral patterns from action histories while the agent plays a game-theoretic optimal (GTO) policy. The second phase progressively shifts the policy toward best-response (BR) exploitation, guided by a per-opponent regularization schedule tied to exploitability. Our architecture introduces dual-turn tokens -- feature vectors constructed at both agent and opponent decision points -- coupled with bucket-rate features that encode opponent tendencies across five strategic contexts. On Leduc Hold'em, a small poker variant with six cards and two betting rounds, we test against six opponent archetypes at two strength levels each, with exploitability ranging from 0.15 to 1.26 Big Blinds (BB) per hand. StratFormer achieves an average exploitation gain of +0.106 BB per hand over GTO, with peak gains of +0.821 against highly exploitable opponents, while maintaining near-equilibrium safety.

Andy Caen, Mark H. M. Winands, Dennis J. N. J. Soemers

Department of Advanced Computing Sciences, Maastricht University, Maastricht, The Netherlands

Opponent Modeling Game-Playing Agents Transformer Imperfect-Information Games Game Theory Multi-Agent Reasoning

Ask PDF

Jul 6, 2026

Predicting Drafted Deck Strength for "Magic: the Gathering"

Many real-world games do not admit a fixed, compact rule set: instead, their dynamics are defined by interactions among a large and often evolving collection of game pieces, making general-purpose policy learning impractical. Magic: the Gathering (MTG) exemplifies this setting, where the cards themselves define and alter gameplay rules, strategic constraints, and long-term outcomes, while the pool of available cards is ever-changing. We study Draft, a constrained deck-building format of MTG in which eight players make 39-45 sequential selections from semi-random packs to construct a 40-card deck under partial information. By isolating the card selection process from gameplay, Draft provides a tractable yet non-trivial setting for studying decision-making driven by combinatorial card synergies. We propose an encoder-based model that produces set-contextualized card embeddings to encode the draft decision sequence, with a consistent improvement over linear baselines on large-scale real-world data, establishing a first learned benchmark for outcome prediction in MTG Draft. Our code is available at github.com/akulen/MtGDraftEncoder.

Tomas Rigaux, Hisashi Kashima

Grad. School of Informatics Kyoto University Kyoto, Japan

Representation Learning

Ask PDF

Abstract

Game-Playing Agents

Explore similar work

Scalable Decision Making for Games of Imperfect Information

StratFormer: Adaptive Opponent Modeling and Exploitation in Imperfect-Information Games

Predicting Drafted Deck Strength for "Magic: the Gathering"