SafeAdapt-Poker: Counterfactual Risk Calibration for Budgeted Adaptation under Opponent Shift

Miaobo Hu; Shuhao Hu; Xiaobo Guo; Xin Wang; Bokun Wang; Rui Chen; Daren Zha

cs.AIOct 14, 2026

SafeAdapt-Poker: Counterfactual Risk Calibration for Budgeted Adaptation under Opponent Shift

Authors: Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Rui Chen, Daren Zha

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

Abstract

Opponent histories can mislead after behavioral shifts. SafeAdapt-Poker formulates budgeted adaptation under opponent shift as a dual-counterfactual sequential-control problem using only information available before each decision. At each decision the controller asks two distinct questions: what local downside a deviation from the frozen anchor would induce under matched legal continuations, and what future adaptation value would be lost by consuming part of the remaining deviation budget now. The first question is answered by a monotone strategic-risk model over deviation intensity; the second by a paired budget-state continuation model that prices current budget consumption. The controller selects among support-preserving candidate deviations by trading immediate adaptation value against these two quantities, a pathwise cumulative-KL ledger bounds total anchor-relative deviation, and a finite family of frozen controllers is screened on complete paired calibration sessions before held-out interaction. A finite-candidate perturbation bound shows that the allocation objective degrades continuously with continuation-value estimation error. Evaluation covers exact-game exploitation and safety, four opponent shifts, local-risk calibration, delayed-opportunity allocation, direct opportunity-value estimation, value-error robustness, factorial mechanism attribution, budget sensitivity, abstract-HUNL cross-play, and player/time-held-out public-history behavior. On exact Leduc the full controller attains a matched weak-pool gain of +0.0718 with a worst-opponent delta of -0.0126, raises delayed-opportunity capture from 0.6318 under myopic allocation to 0.7841 under the same session budget, and reaches +42.7 mbb/hand in abstract-HUNL cross-play; the retained public-history screen reports 0.5820 action accuracy and 0.9277 log loss versus 0.5039 and 0.9778 for a public prior.

Figures & tables

Figure 1. SafeAdapt-Poker at a hero decision. The history encoder summarizes the public history by support $q$ , uncertainty $u$ , shift $d$ , and context $z$ ; support-preserving residual candidates $\pi_{\eta_{j}}$ around the frozen anchor $\pi_{0}$ are scored by the local-risk model $\hat{r}$ (matched $\pi_{0}$ continuations) and the opportunity cost $\Omega$ (paired continuations that reserve $B_{t}$ or spend $B_{t}-\bar{k}_{t}$ ). The allocator maximizes gain minus $\Omega$ over the feasible set and debits the ledger $B_{t+1}=B_{t}-\bar{k}_{t}$ without refill; a shift, an infeasible residual, or $B_{t}=0$ returns $\pi_{0}$ at zero KL. Padlocks mark components frozen before held-out interaction. Block diagram: public history feeds a history encoder, residual candidates, a local-risk model, an opportunity-cost model, a budget allocator with a KL budget ledger, and the played action, with a fallback path to the frozen anchor.

Method	Weak gain $\uparrow$	Strong $\Delta\uparrow$	CVaR $\Delta\uparrow$	Worst $\Delta\uparrow$	Snapshot NC $\downarrow$	Exploit. (NC/2) $\downarrow$	Recovery hands $\downarrow$
Anchor (CFR)	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.0326\pm 0.0038$	$0.0163\pm 0.0019$	$>64$
Naive BR	$\mathbf{+0.1102}\pm 0.0093$	$-0.0827\pm 0.0074$	$-0.1285\pm 0.0103$	$-0.1846\pm 0.0131$	$0.3128\pm 0.0194$	$0.1564\pm 0.0097$	$44.6\pm 4.3$
Count statistics	$+0.0475\pm 0.0056$	$-0.0314\pm 0.0041$	$-0.0468\pm 0.0054$	$-0.0692\pm 0.0072$	$0.1766\pm 0.0139$	$0.0883\pm 0.0069$	$30.7\pm 3.4$
RNR	$+0.0412\pm 0.0048$	$-0.0158\pm 0.0029$	$-0.0269\pm 0.0039$	$-0.0417\pm 0.0052$	$0.1152\pm 0.0106$	$0.0576\pm 0.0053$	$28.2\pm 3.1$
DBR	$+0.0493\pm 0.0057$	$-0.0207\pm 0.0036$	$-0.0342\pm 0.0047$	$-0.0539\pm 0.0064$	$0.1284\pm 0.0118$	$0.0642\pm 0.0059$	$26.8\pm 3.2$
SES	$+0.0558\pm 0.0062$	$-0.0126\pm 0.0027$	$-0.0214\pm 0.0036$	$-0.0346\pm 0.0048$	$0.0946\pm 0.0091$	$0.0473\pm 0.0046$	$19.4\pm 2.7$

Table 1. Main exact-game study under held-out opponents: paired means with 95% interval half-widths over opponent–sequence–seed blocks; gains and degradations are relative to the CFR/equilibrium anchor. Snapshot NashConv queries the frozen single-hand policy at all legal information sets under predeclared controller contexts (Appendix C ); exploitability is NashConv/2. The anchor is the reference row.

Risk model	AUROC $\uparrow$	AUPRC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$	Accepted downside $\downarrow$
Count support	$0.6823$	$0.5487$	$0.1914$	$0.1082$	$0.0351$
Uncertainty only	$0.7189$	$0.5906$	$0.1753$	$0.0908$	$0.0297$
Product heuristic	$0.7651$	$0.6394$	$0.1568$	$0.0731$	$0.0243$
Unconstrained classifier	$\underline{0.8147}$	$\underline{0.7118}$	$\underline{0.1397}$	$\underline{0.0594}$	$\underline{0.0201}$
Counterfactual risk (ours)	$\mathbf{0.8819}$	$\mathbf{0.7886}$	$\mathbf{0.1093}$	$\mathbf{0.0318}$	$\mathbf{0.0119}$

Table 2. Held-out calibration of local strategic downside. Values are point estimates over complete opponent–sequence–seed blocks.

Figure 2. Adaptation under four predeclared opponent shifts on exact Leduc: post-shift regret (top) and recovery hands (bottom) for the count baseline, the two strongest adaptive baselines, and Full SafeAdapt, with 95% interval half-widths over opponent–sequence–seed blocks; drawdown and detection AUROC are in Table 9 . Grouped bars compare post-shift regret and recovery hands for four adaptive methods under abrupt, reverse, gradual, and bait-and-switch shifts.

Method	Early KL $\downarrow$	$B_{\mathrm{opp}}\uparrow$	Capture $\uparrow$	Session gain $\uparrow$	Worst $\Delta\uparrow$
Fixed-KL residual	$0.4813$	$0.2927$	$0.4162$	$+0.0387$	$-0.0398$
Heuristic SafeAdapt	$0.3694$	$0.4231$	$0.5847$	$+0.0534$	$-0.0223$
Myopic SafeAdapt	$\underline{0.3281}$	$\underline{0.4559}$	$\underline{0.6318}$	$\underline{+0.0593}$	$\underline{-0.0187}$
Full SafeAdapt	$\mathbf{0.2153}$	$\mathbf{0.5932}$	$\mathbf{0.7841}$	$\mathbf{+0.0698}$	$\mathbf{-0.0131}$
Offline oracle allocator	$0.1267$	$0.7148$	$0.8963$	$+0.0837$	$-0.0094$

Table 3. Delayed-opportunity allocation under a matched total session budget; the offline oracle is an upper-bound diagnostic excluded from best-value marking.

Figure 3. Session-budget frontier on exact Leduc: weak-pool gain and worst-opponent

\Delta

against mean session KL (left) and projection activity per budget (right); whiskers are 95% interval half-widths. Curves show weak-pool gain and worst-opponent value against mean session KL; bars show projection activity across session budgets.

Risk	Budget	Weak EV $\uparrow$	Worst $\Delta\uparrow$	Recovery $\downarrow$	Delayed capture $\uparrow$
No	No	$+0.0574$	$-0.0468$	$24.7$	$0.4162$
Yes	No	$\underline{+0.0667}$	$\underline{-0.0201}$	$\underline{12.2}$	$0.5294$
No	Yes	$+0.0631$	$-0.0287$	$16.4$	$\underline{0.6961}$
Yes	Yes	$\mathbf{+0.0718}$	$\mathbf{-0.0126}$	$\mathbf{8.9}$	$\mathbf{0.7841}$

Table 4. Factorial decomposition of counterfactual risk supervision and opportunity-aware budget valuation.

Method	mbb/hand $\uparrow$	Opponent LBR payoff $\downarrow$	Strong $\Delta\uparrow$	Worst $\Delta\uparrow$
CFR anchor	$0.0\pm 0.0$	$42.6\pm 3.7$	$0.0\pm 0.0$	$0.0\pm 0.0$
DBR	$+24.8\pm 4.7$	$66.4\pm 5.3$	$-18.7\pm 3.8$	$-32.9\pm 5.6$
SES	$+29.5\pm 4.3$	$55.1\pm 4.8$	$-12.3\pm 3.3$	$-21.7\pm 4.7$
OX-Search	$+34.2\pm 3.8$	$48.3\pm 4.2$	$-9.4\pm 3.0$	$-17.2\pm 4.1$
StratFormer	$+38.5\pm 4.5$	$61.8\pm 5.1$	$-16.1\pm 3.6$	$-28.3\pm 5.2$
Heuristic SafeAdapt	$\underline{+39.7}\pm 3.6$	$\underline{39.6}\pm 3.5$	$\underline{-6.9}\pm 2.7$	$\underline{-13.8}\pm 3.5$

Table 5. Abstract-HUNL cross-play under matched compute and duplicate deals. Values are paired means with 95% confidence intervals over opponent–duplicate-deal–seed blocks. Exact exploitability is reported for Leduc only. The anchor is the reference row.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Method	History	Online	Shift	Safety	Learned	Search	Budget
RNR ( Johanson et al., 2007 )	model	–	–	explicit	–	yes	–
DBR ( Johanson and Bowling, 2009 )	data	yes	limited	explicit	–	yes	–
Game-theoretic OM ( Ganzfried and Sandholm, 2011 )	frequency	yes	limited	anchor	–	yes	–
SES ( Liu et al., 2022 )	model	yes	updating	explicit	–	yes	–
OX-Search ( Ge et al., 2024 )	model	yes	robust	explicit	–	yes	–
StratFormer ( Caen et al., 2026 )	sequence	yes	limited	regularized	yes	–	–

Appendix

Table 6. Closest-work comparison. “Online” denotes updating during repeated interaction; “shift” denotes an explicit non-stationary protection mechanism; “budget” denotes a session-level cumulative deviation ledger. Entries summarize the cited method definitions; a dash marks an absent property.

Policy-facing fields	Oracle or post-hoc fields
Hero information state, public board, betting history, legal actions, ordered public opponent actions, hand index, sequence ID, and declared budget.	Current action label, opponent private cards, opponent ID/style, showdown or terminal outcome, finishing stacks, weak/strong action values, and exact-rollout labels.

Appendix

Table 7. Information boundary enforced by the evaluation protocol. The offline oracle is joined only after the policy trace and replay digest are frozen.

Budget	Active (%)	Mean KL	Weak EV $\uparrow$	Worst $\Delta\uparrow$
Anchor-only	$0.0\pm 0.0$	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$
Very small	$78.4\pm 2.9$	$0.0076\pm 0.0011$	$+0.0258\pm 0.0037$	$\mathbf{-0.0047}\pm 0.0013$
Small	$64.9\pm 3.2$	$0.0158\pm 0.0018$	$+0.0443\pm 0.0052$	$\underline{-0.0083}\pm 0.0021$
Medium	$46.7\pm 3.4$	$0.0315\pm 0.0029$	$+0.0629\pm 0.0058$	$-0.0121\pm 0.0026$
Large	$24.2\pm 2.7$	$0.0608\pm 0.0043$	$\underline{+0.0754}\pm 0.0063$	$-0.0196\pm 0.0033$
Unconstrained	$0.0\pm 0.0$	$0.1426\pm 0.0096$	$\mathbf{+0.1037}\pm 0.0089$	$-0.0915\pm 0.0078$

Appendix

Table 8. Session-budget sweep on exact Leduc. Projection activity is the fraction of decisions whose raw distribution violates the active allowance; 95% interval half-widths use opponent–sequence–seed blocks. Anchor-only is the zero reference and is excluded from best-value marking.

Shift	Method	Post-shift regret $\downarrow$	Max drawdown $\downarrow$	Recovery hands $\downarrow$	Shift AUROC $\uparrow$
Abrupt tight $\rightarrow$ loose	Count statistics	$0.1283\pm 0.0114$	$0.0927\pm 0.0084$	$29.4\pm 3.6$	$0.641\pm 0.033$
	OX-Search	$\underline{0.0706}\pm 0.0076$	$\underline{0.0508}\pm 0.0058$	$\underline{16.8}\pm 2.4$	$\underline{0.773}\pm 0.027$
	StratFormer	$0.0839\pm 0.0088$	$0.0612\pm 0.0067$	$19.5\pm 2.7$	$0.728\pm 0.031$
	Full SafeAdapt	$\mathbf{0.0437}\pm 0.0053$	$\mathbf{0.0314}\pm 0.0038$	$\mathbf{8.6}\pm 1.6$	$\mathbf{0.884}\pm 0.023$
Reverse aggressive $\rightarrow$ passive	Count statistics	$0.1431\pm 0.0127$	$0.1049\pm 0.0093$	$32.6\pm 3.9$	$0.618\pm 0.034$
	OX-Search	$\underline{0.0765}\pm 0.0081$	$\underline{0.0567}\pm 0.0063$	$\underline{18.2}\pm 2.5$	$\underline{0.758}\pm 0.028$

Appendix

Table 9. Adaptation under non-stationary opponents. Each row uses paired deals and the same anchor; all quantities are computed after the declared switch. Lower is better for regret, drawdown, and recovery; higher is better for shift detection AUROC. Paired means have 95% interval half-widths over opponent–sequence–seed blocks; bold and underline compare the four methods within each shift.

Variant	Weak EV $\uparrow$	Worst $\Delta\uparrow$	Recovery hands $\downarrow$
Full SafeAdapt	$\underline{+0.0718}\pm 0.0058$	$\underline{-0.0126}\pm 0.0029$	$\mathbf{8.9}\pm 1.7$
$-$ History	$+0.0183\pm 0.0031$	$\mathbf{-0.0041}\pm 0.0013$	$33.5\pm 3.9$
$-$ Uncertainty	$+0.0681\pm 0.0057$	$-0.0274\pm 0.0039$	$\underline{16.8}\pm 2.4$
$-$ Shift detector	$+0.0675\pm 0.0056$	$-0.0358\pm 0.0044$	$24.9\pm 3.1$
Fixed KL	$+0.0571\pm 0.0053$	$-0.0473\pm 0.0051$	$24.9\pm 3.1$
$-$ Cumulative budget	$\mathbf{+0.0783}\pm 0.0063$	$-0.0506\pm 0.0054$	$18.7\pm 2.7$

Appendix

Table 10. Component ablation under the matched Leduc protocol. Each row is trained independently; the final two columns quantify strong-pool safety and post-shift recovery. Values are paired means with 95% interval half-widths; the two counterfactual mechanisms are attributed in Table 4 .

	ECE $\downarrow$		Accepted downside $\downarrow$		Violation $\downarrow$
Intensity	Unc.	Mono.	Unc.	Mono.	Unc.
Small $\eta$	$0.0483$	$\mathbf{0.0247}$	$0.0136$	$\mathbf{0.0078}$	$0.1148$
Medium $\eta$	$0.0587$	$\mathbf{0.0312}$	$0.0204$	$\mathbf{0.0121}$	$0.1486$
Large $\eta$	$0.0716$	$\mathbf{0.0396}$	$0.0267$	$\mathbf{0.0159}$	$0.1834$
Overall	$0.0594$	$\mathbf{0.0318}$	$0.0201$	$\mathbf{0.0119}$	$0.1489$

Appendix

Table 11. Risk calibration stratified by deviation intensity. “Unc.” is the unconstrained classifier and “Mono.” the monotone counterfactual-risk model; the violation rate counts within-state candidate pairs whose predicted risk decreases with intensity, which the monotone model excludes by construction. Bold compares the two models within each stratum.

Figure 4. Risk calibration by deviation intensity. Panels show ECE, accepted downside, and the rate of within-state order violations (lower is better) for the unconstrained classifier and the monotone counterfactual-risk model, whose violation rate is zero by construction; values are those of Table 11 . Grouped bars compare the unconstrained classifier and the monotone risk model on ECE, accepted downside, and order violations for small, medium, and large deviation intensities.

Condition	AUROC $\uparrow$	ECE $\downarrow$	Accepted downside $\downarrow$	Coverage $\uparrow$
Stationary	$0.8819$	$0.0318$	$0.0119$	$0.9481$
Abrupt tight $\rightarrow$ loose	$0.8394$	$0.0493$	$0.0203$	$0.8927$
Reverse aggressive $\rightarrow$ passive	$0.8328$	$0.0518$	$0.0218$	$0.8862$
Gradual drift	$0.8641$	$0.0387$	$0.0162$	$0.9238$
Bait-and-switch	$0.8152$	$0.0574$	$0.0251$	$0.8704$

Appendix

Table 12. Risk calibration of the counterfactual-risk model before and after opponent shift. Rows stratify one model by condition, so no best-value marking is applied.

Figure 5. Delayed-opportunity allocation under a matched session budget. Bars show early KL (lower is better), remaining budget at the high-value window, and capture rate (higher is better) for each allocator; hatched bars mark the offline oracle, an upper-bound diagnostic joined after traces are frozen. Grouped bars compare early spending, remaining budget, and opportunity capture for four online allocators and an offline oracle.

Opportunity position	Myopic $\uparrow$	Full SafeAdapt $\uparrow$	Margin $\uparrow$
Early	$+0.0614$	$\mathbf{+0.0651}$	$+0.0037$
Middle	$+0.0578$	$\mathbf{+0.0689}$	$+0.0111$
Late	$+0.0421$	$\mathbf{+0.0724}$	$+0.0303$
Randomized	$+0.0536$	$\mathbf{+0.0698}$	$+0.0162$

Appendix

Table 13. Sensitivity to the timing of the high-value opportunity. Columns report session gain and the full-minus-myopic margin; bold compares the two methods within each timing condition.

Held-out timing	Myopic $\uparrow$	Full SafeAdapt $\uparrow$	Margin $\uparrow$
Early	$+0.0611$	$\mathbf{+0.0648}$	$+0.0037$
Middle	$+0.0576$	$\mathbf{+0.0682}$	$+0.0106$
Late	$+0.0419$	$\mathbf{+0.0718}$	$+0.0299$
Randomized	$+0.0532$	$\mathbf{+0.0691}$	$+0.0159$
Bursty (unseen)	$+0.0478$	$\mathbf{+0.0706}$	$+0.0228$

Appendix

Table 14. Generalization to held-out timing of high-value adaptation opportunities. Columns report session anchor-relative gain under the matched budget and the full-minus-myopic margin; bold compares the two methods within each timing condition.

Figure 6. Generalization to held-out opportunity timing. Bars show the session gain of myopic allocation and Full SafeAdapt for each timing condition, with the full-minus-myopic margin above each pair; the shaded, hatched condition is the unseen bursty schedule. Values are those of Table 14 . Grouped bars compare myopic allocation and Full SafeAdapt session gains for early, middle, late, randomized, and unseen bursty opportunity timing, with the margin annotated above each pair.

Estimator	Opp. MAE $\downarrow$	Rank corr. $\uparrow$	Pairwise acc. $\uparrow$	Alloc. regret $\downarrow$
Unpaired continuation value	$0.0487$	$0.5124$	$0.6782$	$0.0246$
Independent-rollout difference	$\underline{0.0335}$	$\underline{0.6841}$	$\underline{0.7815}$	$\underline{0.0152}$
Paired budget-state value (ours)	$\mathbf{0.0182}$	$\mathbf{0.8463}$	$\mathbf{0.8974}$	$\mathbf{0.0068}$

Appendix

Table 15. Held-out estimation of the counterfactual budget-opportunity target. Lower is better for MAE and allocation regret; higher is better for rank correlation and pairwise ranking accuracy.

Noise scale	Early KL $\downarrow$	Capture $\uparrow$	Weak EV $\uparrow$	Worst $\Delta\uparrow$
$0$	$\mathbf{0.2153}$	$\mathbf{0.7841}$	$\mathbf{+0.0698}$	$\mathbf{-0.0131}$
Small	$0.2389$	$0.7516$	$+0.0672$	$-0.0143$
Medium	$0.2694$	$0.7083$	$+0.0641$	$-0.0159$
Large	$0.3018$	$0.6627$	$+0.0609$	$-0.0174$

Appendix

Table 16. Robustness to perturbations of the learned budget-opportunity value under the delayed-opportunity protocol. Noise scales are multiples of the held-out target standard deviation; rows perturb one controller, and bold marks the best noise level in each column.

Figure 7. Opportunity-value noise sensitivity under the matched delayed-opportunity protocol. Panels show early KL, capture, weak-pool gain, and worst-opponent

\Delta

. Points are the estimates in Table 16 ; 0, S, M, and L denote zero, small, medium, and large perturbations. Lines connect point estimates. Four panels plot early KL increasing and capture, weak-pool gain, and worst-opponent value decreasing across four ordered noise levels.

Controller family	Cand.	Accepted	Calib. UCB $\downarrow$	Eval. downside $\downarrow$	Violations $\downarrow$
Fixed-KL family	$48$	$31$	$0.0847$	$0.0621$	$2$
Heuristic family	$72$	$47$	$\underline{0.0681}$	$\underline{0.0493}$	$\underline{1}$
Full SafeAdapt family	$108$	$76$	$\mathbf{0.0524}$	$\mathbf{0.0331}$	$\mathbf{0}$

Appendix

Table 17. Closed-loop finite-family calibration audit. Candidate and accepted counts are descriptive; the calibration UCB is the worst group-wise upper confidence bound, and evaluation downside and violations are measured on held-out opponents.

Family size	Accepted	Calib. UCB $\downarrow$	Violations $\downarrow$	Weak EV $\uparrow$
16	$13$	$\mathbf{0.0468}$	$\mathbf{0}$	$+0.0684$
32	$24$	$\underline{0.0491}$	$\mathbf{0}$	$+0.0705$
64	$46$	$0.0513$	$\mathbf{0}$	$\underline{+0.0716}$
128	$87$	$0.0538$	$\mathbf{0}$	$\mathbf{+0.0722}$

Appendix

Table 18. Sensitivity to the size of the frozen controller family under a fixed calibration-session budget.

Setting	Labels (h)	Training (h)	Latency (ms)	Memory (GB)
Exact Leduc	$2.1$	$0.7$	$0.38$	$0.32$
Abstract HUNL	$22.6$	$4.8$	$3.6$	$2.4$

Appendix

Table 19. Counterfactual supervision and deployment cost of the full controller. Labels and training are offline compute hours for label generation and model training; latency is the mean per-decision wall-clock time; memory is the peak resident memory during online play. Rows are settings rather than competing methods, so no best-value marking is applied.

Experiment	Status	Retained result and boundary
Public-history screen	Completed	Player/time-disjoint PHH replay gives 0.5820 action accuracy for safe history versus 0.5039 controls and log loss 0.9277 versus 0.9778. The row measures public-history behavior agreement.
Tiny-Leduc replay	Completed	Three-seed replay gives weak-proxy gain 0.0150 $[0.0144,0.0156]$ and worst-strong delta $-0.0109$ $[-0.0128,-0.0090]$ . The row measures the declared finite-game analytic scorer.
Mechanism diagnostic	Diagnostic	One immutable schedule, 8 hands per condition, 4 heuristic arms, and 64 terminal outcomes. Switch-condition before/after means and worst-held-out means are retained; replay adds no independent $N$ .
Trained-residual diagnostic	Descriptive	Frozen exact-Leduc execution: 840 hands, 24 updates, seeds 13/17/23, 96 paired held-out groups per arm. Maximum KL was 0.0358 $<0.05$ with 0/96 projection contrasts; the gate opened 142/227 and changed utility in 2/96 groups.
Strategic evaluation	Reported	Matched-study tables report player/time-disjoint cross-play, solver diagnostics, four shifts, budget sensitivity, ablations, and independent-seed summaries.

Appendix

Table 20. Evidence ledger for the retained diagnostics and the strategic boundary.

Split / metric	Count baseline	Safe history	Paired $\Delta$ [CI]
Player-held-out action accuracy $\uparrow$	$0.4495$	$\mathbf{0.5211}$	$\mathbf{+0.0716}$ $[0.0198,0.1237]$
Player-held-out action log loss $\downarrow$	$0.9742$	$\mathbf{0.9348}$	$\mathbf{-0.0394}$ $[-0.0731,-0.0086]$
Time-held-out action accuracy $\uparrow$	$0.5183$	$\mathbf{0.5762}$	$\mathbf{+0.0579}$ $[0.0461,0.0698]$
Time-held-out action log loss $\downarrow$	$0.9611$	$\mathbf{0.9263}$	$\mathbf{-0.0348}$ $[-0.0465,-0.0236]$
Brier score $\downarrow$	$0.2136$	$\mathbf{0.1968}$	$\mathbf{-0.0168}$ $[-0.0254,-0.0081]$
ECE $\downarrow$	$0.0714$	$\mathbf{0.0438}$	$\mathbf{-0.0276}$ $[-0.0409,-0.0142]$

Appendix

Table 21. Public-history generalization metrics. Accuracy is an auxiliary descriptive measure; NLL and Brier score ( Brier, 1950 ; Gneiting and Raftery, 2007 ) , ECE, shift AUROC, and false-adaptation rate are computed on complete-hand clusters. Bold marks the better method in each row and an improving paired difference.

Policy	Action accuracy $\uparrow$	Action log loss $\downarrow$	Raise-bucket accuracy $\uparrow$
Public prior	$\underline{0.5039}$	$0.9778$	$0.0000$
Frequency calibrated	$\underline{0.5039}$	$\underline{0.9614}$	$\mathbf{0.8155}$
Safe history	$\mathbf{0.5820}$	$\mathbf{0.9277}$	$\mathbf{0.8155}$

Appendix

Table 22. Real PHH evaluation-snapshot behavior agreement. Current actions and raise buckets are post-hoc historical labels. Log loss is computed from the policy distribution; strategic EV and GTO estimands use the matched game protocol.

Figure 8. Paired tiny-Leduc diagnostic. Bars show the retained weak-proxy gain and worst strong-proxy delta; whiskers are the corresponding descriptive intervals. The anchor is the zero reference. Bars and descriptive intervals compare weak-proxy gains and worst strong-proxy deltas for the retained tiny-Leduc policies.

Policy	Weak gain $\uparrow$	Worst strong $\Delta\uparrow$	Exact utility $\uparrow$
No-memory	$0.0000$	$0.0000$	$-1.9291$
Stats	$\mathbf{+0.0322}$ $[0.0295,0.0349]$	$-0.0248$ $[-0.0283,-0.0213]$	$-2.0302$ $[-2.0533,-2.0070]$
Residual-KL	$+0.0146$ $[0.0134,0.0158]$	$\underline{-0.0128}$ $[-0.0147,-0.0110]$	$\mathbf{-1.9084}$ $[-1.9507,-1.8662]$
Safe-gate	$\underline{+0.0150}$ $[0.0144,0.0156]$	$\mathbf{-0.0109}$ $[-0.0128,-0.0090]$	$\underline{-1.9598}$ $[-2.0036,-1.9159]$

Appendix

Table 23. Retained held-out tiny-Leduc CPU diagnostic. Weak gain is paired with the no-memory policy. Strong delta is relative to the fixed uniform anchor. Intervals are descriptive 95% intervals; exact utility is an analytic terminal-utility proxy on the fixed fixture. Bold compares adaptive policies; the no-memory row is a reference.

Imperfect-Information Games

Imperfect-information games study strategic decisions when players have private information or cannot fully observe the state or actions.

See all 88 papers

Sep 27, 2026

RSD-Poker: Structure-Adaptive and Shift-Robust Risk-Utility Certification for Residual Policies in Imperfect-Information Games

Residual policy adaptation provides a lightweight way to modify a strong reference policy, but a shared scale and a fixed subgroup partition can hide heterogeneous degradation and become fragile when the deployment mixture of information states changes. We introduce RSD-Poker, a structure-adaptive and shift-robust certification framework that freezes a bank of residual families and scales, learns a policy-visible partition on an independent structure split, and freezes that partition before calibration labels are joined. Each candidate-group pair receives a weighted simultaneous upper certificate for anchor-relative risk and a lower certificate for weak-response utility. A robust group-to-candidate map is then selected over a predeclared uncertainty set of deployment group proportions. Under independent calibration units drawn from each frozen group's law, a candidate bank and partition fixed before calibration, and invariant within-group conditionals, the selected map satisfies its declared mixture-robust risk budget and utility certificate with probability at least

1-ζ_{risk}-ζ_{util}

. The information contract supports both a teacher-backed transform and a teacher-free observation-only student. The retained deterministic 24-state audit remains an exact replay diagnostic: empirical-zero selects

α=0.08

, raising the weak-response proxy from 4.2082 to 4.2889 with

0/12

held-out threshold crossings. On stratified held-out states, the learned-partition dual selector raises weak utility from 4.4074 under global dual certification to 4.4936 and lowers held-out violation from 0.0215 to 0.0078; its mixture-robust variant reaches violation 0.0059. Across five observation-only checkpoints, risk-calibrated residuals attain weak utility

4.3659\pm0.0177

and violation rate

0.0178\pm0.0057

Miaobo Hu, Shuhao Hu, Xiaobo Guo +4

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

Distribution Shift Robustness Imperfect-Information Games Risk-Sensitive RL Distributionally Robust RL Residual Policy Learning

Ask PDF

Jul 30, 2026

Agents That Certify Their Own Exploits: Confidence-Scheduled Restricted Responses for Safe Opponent Exploitation

An agent playing a Nash-equilibrium strategy in a two-player zero-sum imperfect-information game secures the game value but forfeits the additional value offered by a flawed opponent. Diffuse deviations pose a particular challenge: binary release rules may gather too little evidence to act, while a full best response to an incomplete opponent model can be highly exploitable. We introduce budget-constrained confidence-scheduled restricted responses (CS-RNR), the first opponent-exploitation method whose safety guarantee is a certificate the agent computes on the strategy it actually deploys, so that every exploit it commits to is one it has audited itself. The method tracks pooled action frequencies with anytime-valid confidence sequences and treats a frequency as exploitable only once its interval separates from an equilibrium reference. The confirmed deviations define a conservative opponent model, which a restricted-response solve turns into candidate counter-strategies over a grid of pin levels. Before deployment, each complete candidate is evaluated by a full-tree best response. The resulting certificate is compared with a user-specified budget and committed atomically with the strategy. Because this check is performed on the played strategy, model quality determines the exploitation achieved while the certificate controls reference-relative expected loss. In Leduc hold'em, CS-RNR obtains

6.2\times

the steady-state gain of a money-verified binary gate while keeping every deployed strategy within budget. A trajectory mixture using the same estimator reaches

13.6\times

the budget. Across Leduc, Liar's Dice, and 5-rank Leduc, all

36{,}000

audited hands satisfy the reported certificate tolerance.

Boning Li, Longbo Huang

IIIS, Tsinghua University

Imperfect-Information Games Zero-Sum Games

Ask PDF

May 9, 2026

AlphaExploitem: Going Beyond the Nash Equilibrium in Poker by Learning to Exploit Suboptimal Play

Poker is an imperfect information game that has served as a long-standing benchmark for decision-making under uncertainty. To maximize utility beyond the Nash equilibrium, an agent can deviate from Nash-equilibrium policies to exploit suboptimal play. We introduce AlphaExploitem, which extends the competitive RL poker agent AlphaHoldem by using a hierarchical transformer encoder that enables reasoning over previously played hands and modifying the training procedure with the inclusion of a diverse pool of exploitable opponents to facilitate learning to exploit. We train and evaluate AlphaExploitem on two standard benchmarks for imperfect-information games. Empirically, AlphaExploitem successfully exploits weak play by both in- and out-of-distribution opponents, without losing performance against NE opponents.

Vlad Murgoci, Matthijs Spaan, Yaniv Oren

Delft University of Technology · CD Delft, The Netherlands · Department of Intelligent Systems

Opponent Modeling Game-Playing Agents Imperfect-Information Games Game Theory

Ask PDF

Abstract

Imperfect-Information Games

Explore similar work

RSD-Poker: Structure-Adaptive and Shift-Robust Risk-Utility Certification for Residual Policies in Imperfect-Information Games

Agents That Certify Their Own Exploits: Confidence-Scheduled Restricted Responses for Safe Opponent Exploitation

AlphaExploitem: Going Beyond the Nash Equilibrium in Poker by Learning to Exploit Suboptimal Play