Climate Surrogates for Scalable Multi-Agent Reinforcement Learning: A Case Study with CICERO-SCM

Oskar Bohn Lassen; Serio Angelo Maria Agriesti; Filipe Rodrigues; Blaz Kurnik; Francisco Camara Pereira

cs.LGOct 9, 2025

Climate Surrogates for Scalable Multi-Agent Reinforcement Learning: A Case Study with CICERO-SCM

Authors: Oskar Bohn Lassen, Serio Angelo Maria Agriesti, Filipe Rodrigues, Blaz Kurnik, Francisco Camara Pereira

Organizations: Technical University of Denmark Kongens Lyngby, Denmark

Abstract

Climate policy analysis requires models that capture multi-gas climate effects, but such models are too slow to embed in reinforcement learning loops at scale. In collaboration with the European Environment Agency, we develop a multi-agent reinforcement learning (MARL) framework that integrates a higher-fidelity climate surrogate as the environment transition, enabling regional agents to learn policies under multi-gas dynamics. We train a recurrent surrogate on $20{,}000$ multi-gas emission pathways to emulate CICERO-SCM. The surrogate achieves near-simulator accuracy (global-mean temperature RMSE $\approx\!4\!\times \!10^{-4}\,\mathrm{K}$ ) with $\sim\!1000\times$ faster one-step inference and yields $>\!100\times$ end-to-end MARL training speed-up. We show policy agreement with the simulator in tractable settings and propose a replay- and rank-consistency test (Kendall's $τ$ ) for assessing policy fidelity when simulator-in-the-loop training is infeasible. This enables large-scale multi-agent policy experiments while retaining high-fidelity multi-gas climate response.

Figures & tables

Figure 1 : Framework for integrating climate surrogates into MARL. In Module 1, agents choose policies that result in emissions, which in Module 2 are mapped into temperature change by a pretrained surrogate and in Module 3 converted into costs.

Figure 2 : Global mean surface air temperature change for the generated emission trajectories.

Climate engine	Test data (RMSE, $R^{2}$ )	Mean inference [s] (CPU/GPU)	Speed-up (CPU/GPU)	Scenario (i) (RMSE, rank- $\tau$ )	Scenario (ii) (RMSE, rank- $\tau$ )	Mean env-step [s]	Speed-up
	Test data performance & inference speed			Policy-induced performance & MARL speed
CICERO–SCM	–	$464.4\times 10^{-3}\,$ / –	–	–	–	$217.7\times 10^{-3}\,$	–
LSTM	( $4.7\times 10^{-4}\,$ , 0.99)	$1.1\times 10^{-3}\,$ / $0.4\times 10^{-3}\,$	$442{\times}$ / $1161{\times}$	( $5.9\times 10^{-4}\,$ , 0.996)	( $3.2\times 10^{-4}\,$ , 0.990)	$1.6\times 10^{-3}\,$	$137{\times}$
GRU	( $3.7\times 10^{-4}\,$ , 0.99)	$2.3\times 10^{-3}\,$ / $0.4\times 10^{-3}\,$	$202{\times}$ / $1161{\times}$	( $3.9\times 10^{-4}\,$ , 0.996)	( $2.0\times 10^{-4}\,$ , 0.997)	$1.6\times 10^{-3}\,$	$137{\times}$
TCN	( $6.8\times 10^{-4}\,$ , 0.99)	$3.3\times 10^{-3}\,$ / $1.3\times 10^{-3}\,$	$140{\times}$ / $357{\times}$	( $21.1\times 10^{-4}\,$ , 0.994)	( $10.3\times 10^{-4}\,$ , 0.982)	$4.5\times 10^{-3}\,$	$49{\times}$

Table 1 : Surrogate performance on held-out test data and policy-induced trajectories and acceleration of inference speed and MARL environment step in scenario (i). Speed-up is measured relative to CICERO-SCM.

Figure 3 : Comparison of learned policies in tractable scenario (i) between CICERO-SCM and the GRU-based surrogate.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Species	Class	Forcing Sign	Model Treatment
CO 2 (FF)	Long-lived GHG	Warming	Carbon-cycle
CO 2 (AFOLU)	Long-lived GHG	Warming	Carbon-cycle
CH 4	Short-lived GHG	Warming	Simplified decay (multi- $\tau$ )
N 2 O	Long-lived GHG	Warming	Fixed lifetime decay
SO 2	Aerosol precursor	Cooling	Linear forcing proxy
CFC-11	Long-lived GHG	Warming	Fixed lifetime decay

Appendix

Table A.1 : CICERO-SCM species grouped by class and defined by atmospheric lifetime, forcing sign, and model treatment.

Group	Parameter	Value	Description
Climate response parameters	rlamdo	15.0836	Air–sea heat exchange parameter [W m -2 K -1 ]
	akapa	0.6568	Vertical heat diffusivity [cm 2 s -1 ]
	cpi	0.2077	Polar amplification factor
	W	2.2059	Upwelling velocity [m yr -1 ]
	beto	6.8982	Ocean heat exchange coefficient [W m -2 K -1 ]
	lambda	0.6063	Climate sensitivity parameter [K W -1 m 2 ]

Appendix

Table A.2 : Baseline parameter configuration used in CICERO-SCM. Parameters are grouped into those governing the climate system response (top) and those scaling emissions into concentrations and effective radiative forcing (bottom).

Panel of Figure 7 — Figure A.1 : Ensemble of generated emission trajectories for the five controllable gases. Shaded regions represent the 5–95% range across the 20,000 generated scenarios, solid lines indicate the ensemble median, dashed lines mark the SSP2-4.5 baseline.

Panel of Figure 8 — Figure A.2 : Comparison of temperature trajectories simulated by CICERO-SCM (solid) and the RNN-based surrogates (dashed) for randomly selected sequences in the test data. Each panel shows results for one architecture (GRU, LSTM, and TCN), illustrating the perfect agreement between surrogate predictions and the ground-truth simulator.

Item	Specification
Agents ( $N$ )	$N=4$ agents with shares $S_{i}=[0.25,\,0.25,\,0.25,\,0.25]$
Climate engines	CICERO-SCM and surrogates
Controlled gases	CO2_FF, CO2_AFOLU, CH4, N2O, SO2
Levers and levels
Energy	Levels: $0.0,\ 0.5,\ 1.0$
Methane	Levels: $0.0,\ 0.5,\ 1.0$

Appendix

Table A.3: Parameters used in MARL scenario (i).

Item	Specification
Agents ( $N$ )	$N=10$ agents with shares $S_{i}=[0.35,0.15,0.10,0.05,0.02,0.01,0.03,0.14,0.1,0.05]$
Climate engines	Surrogates only
Controlled gases	CO2_FF, CO2_AFOLU, CH4, N2O, SO2
Levers and levels
Energy	Levels: $0.0,\ 0.5,\ 1.0$
Methane	Levels: $0.0,\ 0.5,\ 1.0$

Appendix

Table A.4: Parameters used in MARL scenario (ii).

Topic	Scenario	Description of figure
Training time comparison	Homogeneous	Fig. A.3 — Wall-clock training time for(surrogate vs. simulator)
Reward convergence	Homogeneous	Fig. A.4 — Reward convergence per agent (surrogate vs. simulator)
Reward convergence	Heterogeneous	Fig. A.5 — Reward convergence per agent (surrogate only)
Mean lever policies	Homogeneous	Fig. A.6 — Mean lever convergence (surrogate vs. simulator)
Mean lever policies	Heterogeneous	Fig. A.7 — Mean lever efforts convergence (surrogates only)
Per-agent levers (GRU)	Homogeneous	Fig. A.8 — Per-agent lever convergence (SCM vs. GRU)

Appendix

Table A.5: Overview of additional figures for MARL experiments.

Panel of Figure 12 — Figure A.3 : Comparison of training time during MARL for the three surrogate architectures (GRU, LSTM, and TCN) relative to CICERO-SCM in the homegeneous scenario (i). Each panel shows the wall-clock training time (log scale) and the corresponding speed-up achieved by replacing the simulator with the surrogate model.

Panel of Figure 13 — Figure A.4 : Comparison of training reward per agent for the three surrogate architectures (GRU, LSTM, and TCN) relative to CICERO-SCM in the homogeneous scenario (i). Each panel shows the evolution of agents’ rewards across environment steps, comparing trajectories obtained with the surrogate (dashed) and the simulator (solid).

Panel of Figure 14 — Figure A.5 : Training reward per agent in the heterogeneous scenario (ii) for GRU, LSTM, and TCN surrogates. Each panel shows reward trajectories over environment steps.

Panel of Figure 15 — Figure A.6 : Comparison of per-lever mean policy trajectories in the homogeneous scenario (i) between CICERO-SCM and the surrogate models (GRU, LSTM, and TCN). Each line shows the evolution of average lever efforts across agents across episodes for both engines, with solid lines representing CICERO-SCM and dashed lines the corresponding surrogate. The close alignment indicates that the surrogates reproduce the learned policy dynamics of the simulator.

Panel of Figure 16 — Figure A.7 : Per-lever mean policy trajectories in the heterogeneous scenario (ii) for the GRU, LSTM, and TCN surrogates. Each line shows the evolution of average lever efforts across agents across episodes.

Figure A.8 : Trajectories of per-agent mean lever effect across episodes shown across environment steps for the homogeneous scenario (i) under CICERO-SCM and GRU surrogate.

Figure A.9 : Trajectories of per-agent mean lever effect across episodes shown across environment steps for the homogeneous scenario (i) under CICERO-SCM and LSTM surrogate.

Figure A.10 : Trajectories of per-agent mean lever effect across episodes shown across environment steps for the homogeneous scenario (i) under CICERO-SCM and TCN surrogate.

Figure A.11 : Trajectories of per-agent mean lever effect for heterogeneous scenario (ii) across episodes shown across environment steps under CICERO-SCM and GRU surrogate.

Figure A.12 : Trajectories of per-agent mean lever effect for heterogeneous scenario (ii) across episodes shown across environment steps under CICERO-SCM and LSTM surrogate.

Figure A.13 : Trajectories of per-agent mean lever effect for heterogeneous scenario (ii) across episodes shown across environment steps under CICERO-SCM and TCN surrogate.

Explore similar work

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
Aug 12, 2026Simon Yu, Nicholas Tomlin, Marwa Abdulhai +7Multi-Agent Simulations Simulation-Based Reinforcement Learning
1Northeastern University · 2New York University · 3UC Berkeley +2
Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Multi-Agent Reinforcement Learning Flow Policies
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics
May 28, 2026Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders JonssonMulti-Agent Reinforcement Learning Structural Constraints
Department of Engineering, University Pompeu Fabra

Aug 12, 2026, cs.CLJ/K move · Enter open · S save

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu+6

1Northeastern University · 2New York University · 3UC Berkeley+2

Multi-Agent Simulations Simulation-Based Reinforcement Learning Multi-Agent Reinforcement Learning Human-Ai Interaction Collapse Diversity

Open paper AskPDF arXiv

Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good,

τ^2

-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

Abstract

Explore similar work

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics