DeepWaveRL: Self-Supervised Full Waveform Inversion via Reinforcement Learning

Peng Jin; Yinan Feng; Yinpeng Chen; Youzuo Lin; Sharon Huang

cs.LGNov 2, 2026

DeepWaveRL: Self-Supervised Full Waveform Inversion via Reinforcement Learning

Authors: Peng Jin, Yinan Feng, Yinpeng Chen, Youzuo Lin, Sharon Huang

Organizations: University of North Carolina, Chapel Hill, U.S. · Google DeepMind, U.S. · Pennsylvania State University, U.S.

Abstract

Full Waveform Inversion (FWI) is a fundamental inverse problem in geophysics to estimate subsurface properties, such as velocity, from seismic measurements. Recently, self-supervised learning methods remove the need for ground-truth velocity maps during training by leveraging a forward operator to simulate seismic data from predictions and backpropagate the gradients of reconstruction loss through it. In this work, we consider a more challenging setting without accessing such gradients. To this end, we incorporate reinforcement learning into self-supervised FWI and propose a framework named DeepWaveRL. Specifically, we reformulate FWI as a policy learning problem, where the policy network predicts velocity maps as actions, and a forward operator is used solely for simulation. A global scalar reward is computed based on data misfit for each prediction and used to weight policy gradients. Thus, learning proceeds without accessing the gradients from the forward operator. Given the indirect and coarse supervision signals, we further identify key strategies to stabilize learning and mitigate the spatial credit assignment bottleneck. Experimental results show that, despite relying entirely on a single scalar reward, DeepWaveRL achieves competitive performance compared to supervised and self-supervised baselines. These findings suggest that our framework offers a flexible and complementary learning paradigm for the FWI research community.

Figures & tables

Figure 1: Comparison between data-driven FWI methods. Top : Supervised learning methods compute loss between predicted and ground-truth velocities; Middle : Exact-gradient self-supervised learning method computes the loss between input and reconstructed seismic data and backpropagates gradients through a forward operator $f$ ; Bottom : Our DeepWaveRL uses the forward operator solely to compute a global scalar reward based on data misfit, without accessing gradients from $f$ .

Figure 2: Schematic illustration of the policy optimization pipeline in DeepWaveRL. For brevity, we only show the seismic data from one source.

Method	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
InversionNet	0.0409	0.0944	0.8796
UPFWI	0.0739	0.1369	0.7420
DeepWaveRL	0.0717	0.1300	0.8303
DeepWaveRL + TTO	0.0527	0.1012	0.8601

Table 1: Quantitative results evaluated on CVA. For models with discrete predictions, we report the mean estimate.

Figure 3: Illustration of ground truth and inversion results of different methods on CVA.

Dataset	Method	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
FFA	InversionNet	0.0098	0.0276	0.9880
	UPFWI	0.0214	0.0484	0.9674
	DeepWaveRL	0.0301	0.0557	0.9062
	DeepWaveRL + TTO	0.0268	0.0476	0.9146
CFA	InversionNet	0.0164	0.0480	0.9721
	UPFWI	0.0280	0.0641	0.9492

Table 2: Quantitative results evaluated on FFA and CFA.

Figure 4: Illustration of ground truth and inversion results of different methods on FFA (top) and CFA (bottom).

Setting	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
Continuous Action Space	0.1443	0.2073	0.4359
Discrete Action Space	0.0527	0.1012	0.8601

Table 3: Quantitative results evaluated on CVA, with different choices of action space.

Figure 5: Illustration of ground truth and inversion results of DeepWaveRL with different choices of action space

Dataset	Fine-tuning	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
FFA	✓	0.0268	0.0476	0.9146
FFA		0.0287	0.0520	0.9347
CFA	✓	0.0278	0.0502	0.9323
CFA		0.0358	0.0647	0.9153

Table 4: Comparison of DeepWaveRL+TTO with and without fine-tuning on the training set of FFA and CFA.

Table 10

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	CVA		FFA & CFA
Test-time Optimization		$\checkmark$		$\checkmark$
Training Steps	44,880	1,440	7,360	1,600
Initial Learning Rate	8e-4	1.6e-4	6.4e-4	6.4e-4
Learning Rate Decay	/	1,360	/	/
Batch Size	128	2048	2048	2048
Group Size	256	16	32	32

Appendix

Table 7: Training details

Method (Hardware)	CVA	FFA/CFA
InversionNet (1xH100)	51 mins	1 hr 30 mins
UPFWI (4xH100)	14 hrs 45 mins	27 hrs
DeepWaveRL (16xH100)	27 hrs	11 hrs

Appendix

Table 8: Wall-clock training time.

Method	Mean	Std
DeepWaveRL	0.3773	0.8622
UPFWI	0.3975	0.9789
InversionNet	0.3381	0.9418

Appendix

Table 9: Wall-clock inference time (single pass inference).

Dataset	Epoch	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
CVA	720	0.0527	0.1012	0.8601
	360	0.0569	0.1075	0.8524
	180	0.0676	0.1172	0.8455
FFA	800	0.0268	0.0476	0.9146
	400	0.0270	0.0471	0.9143
	200	0.0277	0.0487	0.9134

Appendix

Table 10: Comparison of DeepWaveRL+TTO with different computation budgets.

Figure 6: Comparison of DeepWaveRL+TTO with different computation budgets on CVA (top row), FFA (middle row) and CFA(bottom row).

Figure 7: Illustration of ground truth and inversion results of SPSA and DeepWaveRL+TTO on CVA

Dataset	Method	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
FFA	InversionNet	0.0087	0.0252	0.9893
	UPFWI	0.0200	0.0429	0.9738
	DeepWaveRL+TTO	0.0268	0.0476	0.9146
CFA	InversionNet	0.0155	0.0467	0.9731
	UPFWI	0.0300	0.0895	0.9514
	DeepWaveRL+TTO	0.0278	0.0502	0.9323

Appendix

Table 11: Comparison of different models transferred from CVA.

Dataset	Method	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
CVA	UPFWI	0.0739	0.1369	0.7420
	UPFWI+TTO	0.0666	0.1243	0.7634
	DeepWaveRL+TTO	0.0527	0.1012	0.8601
FFA	UPFWI	0.0214	0.0484	0.9676
	UPFWI+TTO	0.0176	0.0364	0.9778
	UPFWI+TTO (CVA init.)	0.0167	0.0331	0.0.9800

Appendix

Table 12: Comparison of UPFWI with and without TTO (and with transfer learning from CVA) and DeepWaveRL+TTO.

$\sigma^{2}(\times 10^{-3})$	PSNR	MAE $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
0	/	0.0527	0.1012	0.8601
1	30.00	0.0593	0.1104	0.8446
5	23.01	0.0907	0.1504	0.7931
10	20.00	0.1358	0.2090	0.7418

Appendix

Table 13: Performance of DeepWaveRL+TTO tested on seismic data with different noise levels.

Figure 8: Comparison of DeepWaveRL+TTO tested on seismic data with different noise levels.

Figure 9: Illustration of ground truth and inversion results of different methods on CVA.

Figure 10: Illustration of ground truth and inversion results of different methods on FFA and CFA.

Figure 11: Comparison of DeepWaveRL+TTO with different numbers of bins on CVA.

Figure 12: Illustration of ground truth and inversion results of DeepWaveRL with and without the sign-preserving logarithm transformation

Figure 13: Comparison of DeepWaveRL with different group sizes on CVA.

Figure 14: Comparison of DeepWaveRL+TTO with and without refining the prior on the training set of FFA (top two rows) and CFA (bottom two rows).

Figure 15: Examples of failure cases where the predictions collapse to similar patterns.

Figure 16: Examples of predicted velocity maps generated by the model that has experienced unstable training.

Inverse Problems

Inverse problems infer hidden causes, parameters, or signals from indirect, noisy measurements using physical models, priors, and statistical methods.

See all 258 papers

Self-Flow-Matching assisted Full Waveform Inversion
Mar 13, 2026Xinquan Huang, Paris PerdikarisInverse Problems Physics-Informed Generative Modeling
University of Pennsylvania
Seismic full-waveform inversion based on a physics-driven generative adversarial network
Mar 16, 2026Xinyi Zhang, Caiyun Liu, Jie Xiong +1Seismic Imaging Inverse Problems
School of Electronic Information and Electrical Engineering, Yangtze University Jingzhou, Hubei 434023, P.R.China
Full Waveform Inversion with a Learned Preconditioner
Oct 11, 2026Jeongjin ParkSeismic Imaging Inverse Problems

Mar 13, 2026J/K move · Enter open · S save

Self-Flow-Matching assisted Full Waveform Inversion

Xinquan Huang, Paris Perdikaris

University of Pennsylvania

Inverse Problems Physics-Informed Generative Modeling Flow Matching

Open paper AskPDF arXiv

Full-waveform inversion (FWI) is a high-resolution seismic imaging method that estimates subsurface velocity by matching simulated and recorded waveforms. However, FWI is highly nonlinear, prone to cycle skipping, and sensitive to noise, particularly when low frequencies are missing or the initial model is poor, leading to failures under imperfect acquisition. Diffusion-regularized FWI introduces generative priors to encourage geologically realistic models, but these priors typically require costly offline pretraining and can deteriorate under distribution shift. Moreover, they assume Gaussian initialization and a fixed noise schedule, in which it is unclear how to map a deterministic FWI iterate and its starting model to a well-defined diffusion time or noise level. To address these limitations, we introduce Self-Flow-Matching assisted Full-Waveform Inversion (SFM-FWI), a physics-driven framework that eliminates the need for large-scale offline pretraining while avoiding the noise-level alignment ambiguity. SFM-FWI leverages flow matching to learn a transport field without assuming Gaussian initialization or a predefined noise schedule, so the initial model can be used directly as the starting point of the dynamics. Our approach trains a single flow network online using the governing physics and observed data. At each outer iteration, we build an interpolated model and update the flow by backpropagating the FWI data misfit, providing self-supervision without external training pairs. Experiments on challenging synthetic benchmarks show that SFM-FWI delivers more accurate reconstructions, greater noise robustness, and more stable convergence than standard FWI and pretraining-free regularization methods.

Abstract

Inverse Problems

Explore similar work

Self-Flow-Matching assisted Full Waveform Inversion

Seismic full-waveform inversion based on a physics-driven generative adversarial network

Full Waveform Inversion with a Learned Preconditioner