Opponent State Inference Under Partial Observability: An HMM-POMDP Framework for 2026 Formula 1 Energy Strategy

Kalliopi Kleisarchaki

cs.AIMar 1, 2026

Opponent State Inference Under Partial Observability: An HMM-POMDP Framework for 2026 Formula 1 Energy Strategy

Authors: Kalliopi Kleisarchaki

Organizations: Independent Researcher

Abstract

The 2026 Formula 1 technical regulations introduce a fundamental change to energy strategy: under a 50/50 internal combustion engine / battery power split with unlimited regeneration and a driver-controlled Override Mode, the optimal energy deployment policy depends not only on a driver's own state but on the hidden state of rival cars. This creates a Partially Observable Stochastic Game that cannot be solved by single-agent optimisation methods. We present a tractable two-layer inference and decision framework. The first layer is a 40-state Hidden Markov Model (HMM) that infers a probability distribution over each rival's ERS charge level (four modes: H, M, L_harvest, L_derate), Override Mode status, and tyre degradation state from six publicly observable telemetry signals. The second layer is a Deep Q-Network (DQN) policy that takes the HMM belief state as input and selects between energy deployment strategies. We formally characterise the counter-harvest trap, a deceptive strategy in which a car deliberately suppresses observable deployment signals to induce a rival into a failed attack, and show that detecting it requires belief-state inference over both ERS level and the harvest/derate sub-mode. On synthetic races, the HMM achieves 96.8% ERS-level accuracy (random baseline 25%), classifies L_harvest vs. L_derate with 89.4% accuracy, and detects counter-harvest trap conditions with 96.3% recall. Pre-season analysis indicates circuit-dependent recharge availability (1.0x to 2.2x per lap) as the primary confound; Melbourne is the hardest-case validation environment. Baum-Welch calibration on 2026 race telemetry begins with the Australian Grand Prix (8 March 2026).

Figures & tables

ERS mode	MOM	$\mu$	$\sigma$	Interpretation
$H$	available	$+4.0$	$1.5$	Full deploy
$H$	spent	$+3.5$	$1.5$	Post-Override, slight ers residual
$M$	any	$+1.0$	$1.5$	Moderate deployment
$L_{\mathrm{harvest}}$	any	$-2.0$	$1.5$	Deliberate harvest; managed throttle
$L_{\mathrm{derate}}$	any	$-1.5$	$1.5$	SOC ceiling; slightly less depressed than harvest
MOM active (transient): $\mu=+7.0$ , $\sigma=1.0$

Table 1:

\Delta v_{\mathrm{trap}}

emission parameters. Non-activation zones.

ERS state	Sub-mode	$\mu$	$\sigma$	Interpretation
$H$	—	$0.05$	$0.04$	Battery not depleted; rarely clips
$M$	—	$0.15$	$0.08$	Occasional clipping on long straights
$L_{\mathrm{harvest}}$	deliberate	$0.08$	$0.05$	Managed throttle; low clipping
$L_{\mathrm{derate}}$	SOC ceiling	$0.55$	$0.18$	Full throttle, SOC ceiling hit

Table 2:

\delta_{\mathrm{throttle}}

emission parameters. Values are Gaussian

(\mu,\sigma)

truncated to

[0,1]

Component	Encoding	Dims	Notes
$e_{\mathrm{self}}$	one-hot	4	Own ERS level (4 modes in v2)
$m_{\mathrm{self}}$	one-hot	2	Own Override Mode status
$f_{\mathrm{self}}$	one-hot	3	Fuel (from lap number)
$\mathrm{gap\_ahead}$	one-hot	4	close / threat / neutral / clear
$\mathrm{gap\_behind}$	one-hot	4
$\delta_{\mathrm{gap}}$	scalar	1	Closure rate, normalised $[-1,1]$

Table 3: DQN input vector components. Total: 66 dimensions (increased from 55 in v1/v1.5 due to expansion of the belief vector from 30 to 40 states and expansion of own ers encoding from 3 to 4 modes).

Metric	HMM	Random	Notes
Top-1 state accuracy	49.1%	2.5%	Full 40-state identification
ERS level accuracy	96.8%	25.0%	4-class; primary inference target
$L_{\mathrm{harvest}}$ vs. $L_{\mathrm{derate}}$	89.4%	50.0%	Key v2 capability
Override status accuracy	61.2%	50.0%	Proximity-conditioned
Tyre cliff detection	91.3%	20.0%	Step-change signal
Trap detection recall	96.3%	—	117 trap sectors; 3.7% miss rate

Table 4: HMM inference accuracy on synthetic races (pre-calibration, v2 with 40 states and

\delta_{\mathrm{throttle}}

). Random baselines: top-1 = 2.5%, ERS-level = 25.0% (4-class), MOM = 50.0%, tyre = 20.0%, harvest/derate = 50.0%.

Perturbation	ERS Accuracy	Change from Baseline
Mean shift $-20\%$	76.1%	$-$ 7.3 pp
Mean shift $-10\%$	80.8%	$-$ 2.6 pp
Baseline (0%)	83.4%	—
Mean shift $+10\%$	87.6%	$+$ 4.0 pp
Mean shift $+20\%$	89.3%	$+$ 5.5 pp
Noise inflation $+10\%$	81.1%	$-$ 2.5 pp

Table 5: ERS-level accuracy under emission parameter misspecification.

Threshold (km/h)	Trap Recall	False Positive Rate
1.0	97.9%	34.2%
1.5	95.1%	23.1%
2.0	91.4%	13.7%
2.5	85.2%	7.6%
3.0	75.3%	3.9%
3.5	63.1%	1.8%

Table 6: Trap recall and false positive rate vs. Active Aero detection threshold.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Observable	Bins	Boundaries	Notes
$\Delta v_{\mathrm{trap}}$ (km/h)	8	$(-\infty,-8,-4,-1,1,3,5,8,+\infty)$
$\Delta t_{\mathrm{sector}}$ (s)	7	$(-\infty,-0.6,-0.2,0.2,0.6,1.0,1.5,+\infty)$
$\Delta b_{\mathrm{brake}}$ (m)	5	$(-\infty,-8,-3,3,10,+\infty)$
$\sigma^{2}_{\mathrm{speed}}$	5	$(-\infty,0.06,0.12,0.18,0.24,+\infty)$
$z_{\mathrm{aero}}$	2	$\{0,1\}$	Bernoulli
$\delta_{\mathrm{throttle}}$	6	$[0,0.05,0.15,0.30,0.50,0.75,1.0]$	Truncated Gaussian

Appendix

Table 7: Discretisation bin boundaries for all six observables.

	$T^{\mathrm{burn}}$				$T^{\mathrm{harvest}}$
	$H$	$M$	$L_{H}$	$L_{D}$	$H$	$M$	$L_{H}$	$L_{D}$
$H$	0.60	0.35	0.03	0.02	0.95	0.05	0.00	0.00
$M$	0.05	0.55	0.25	0.15	0.60	0.35	0.03	0.02
$L_{\mathrm{harvest}}$	0.00	0.25	0.50	0.25	0.30	0.55	0.15	0.00
$L_{\mathrm{derate}}$	0.00	0.05	0.15	0.80	0.00	0.45	0.30	0.25

Appendix

Table 8: ERS transition matrices

T^{\mathrm{burn}}

(left) and

T^{\mathrm{harvest}}

(right). The

L_{\mathrm{harvest}}

and

L_{\mathrm{derate}}

rows differ materially under burn: a harvesting car has reserve to deploy; a derating car does not.

Hyperparameter	Value
Network architecture	66-256-256-128-2
Activation	ReLU
Weight initialisation	He (Kaiming normal)
Replay buffer capacity	50,000 transitions
Batch size	64
Discount factor $\gamma$	0.95

Appendix

Table 9: DQN training hyperparameters.

Profile	$p_{\mathrm{burn}}$ schedule	Trap execution	Notes
Aggressive	0.75 throughout	Never	Consistently high energy use
Conservative	0.35 throughout	Never	Sustained harvesting
Trap-setter	0.30 baseline; 0.90 post-trap	In activation zones when gap $<1.5$ s	Primary test of Proposition 1; operates from $L_{\mathrm{harvest}}$
Balanced	0.55 throughout	Never	Average behaviour

Appendix

Table 10: Synthetic strategy profile definitions.

Partially Observable Stochastic Games

See all 14 papers

Feb 26, 2026

Learning-based Multi-agent Race Strategies in Formula 1

In Formula 1, race strategies are adapted according to evolving race conditions and competitors' actions. This paper proposes a reinforcement learning approach for multi-agent race strategy optimization. Agents learn to balance energy management, tire degradation, aerodynamic interaction, and pit-stop decisions. Building on a pre-trained single-agent policy, we introduce an interaction module that accounts for the behavior of competitors. The combination of the interaction module and a self-play training scheme generates competitive policies, and agents are ranked based on their relative performance. Results show that the agents adapt pit timing, tire selection, and energy allocation in response to opponents, achieving robust and consistent race performance. Because the framework relies only on information available during real races, it can support race strategists' decisions before and during races.

Giona Fieni, Joschua Wüthrich, Marc-Philippe Neumann +1

Autonomous Racing Multi-Agent Reinforcement Learning

Ask PDF

Oct 13, 2026

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent's intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density.

Marcel Petran, Stefan Edelkamp

Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

Opponent Modeling Multi-Agent Reinforcement Learning Partially Observable RL Partially Observable Stochastic Games Deep Q-Learning

Ask PDF

Jun 25, 2026

State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.

Jesper Klicks, Sander Vržina, Vincent François-Lavet

State Representation Learning Power Systems

Ask PDF

Abstract

Partially Observable Stochastic Games

Explore similar work

Learning-based Multi-agent Race Strategies in Formula 1

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading