Learning spatially varying regularisation parameters of low regularity for image reconstruction
Authors: Kostas Papafitsoros, Luca Calatroni, Andreas Kofler
Organizations: School of Mathematical Sciences, Queen Mary University of London, London E1 4BS, United Kingdom · MaLGa centre, DIBRIS, Universit`a di Genova, Istituto Italiano di Tecnologia, Genoa, Italy · Physikalisch-Technische Bundesanstalt (PTB), Braunschweig and Berlin, Germany
In this chapter, we review and discuss the regularity properties of spatially adaptive regularisation weight functions used in variational image reconstruction. Incorporating such weights into classical model-based regularisers, such as Total Variation (TV) and Total Generalised Variation (TGV), allows the regularisation strength to vary across the image and adapt to local image content. When appropriately estimated, these weights can thus significantly improve edge and detail preservation in the reconstructions. We review the existing theoretical literature on this topic for different regularity classes, including constant, continuous, and piecewise constant functions. Our discussion is motivated by recent work on hybrid image reconstruction methods that combine model-based regularisation with deep neural networks to learn highly adaptive regularisation weights. In particular, we discuss how the structural properties of these weights influence the reconstruction from both theoretical and practical perspectives. Through representative examples in image denoising and magnetic resonance imaging (MRI) reconstruction, we demonstrate that the learned weights are often of low regularity and can adapt not only to the image structure but also to the specific noise realisation. We conclude by highlighting several directions for future research on this topic.
Figures & tables
Figure 1: Intensity values for a grayscale digital image over a pixel grid, where a value, typically in [0,1] , represents the brightness of the pixel (0 for black, 1 for white). We regard these images as discretised versions of functions from a domain Ω⊂R2→R .
Figure 2: Comparison of classical derivative-based regularisers for a Gaussian denoising task. Tikhonov regularisation produces smooth outputs, destroying the edges; TV preserves edges but also leads to staircasing artifacts, while TGV both preserves edges and eliminates the staircasing artifacts.
Figure 3: Top: TV denoising with a scalar λ for a simple affine function f . Middle: Example of creation of a new discontinuity in the solution of the TV denoising problem with spatially varying weight Λ , at a point x=0.5 where DΛ′({x})>0 . Bottom: Creation of a flat area around the point x=0.5 where DΛ′({x})<0 .
Figure 4: Example of dyadic partitions of Ω=(0,1)2 considered in [ 21 ] leading to discontinuous but structured regularisation weights Λ . For comparison, we depict again a low-regularity weight computed by a neural network, see Section 4.2 for details.
Figure 5: Example of a data function f and utrue:=uΛ , which is solution of ( ^ P ) with zero objective but cannot be a solution of the problem ( P ).
Figure 6: Visualisation of the unrolled network NθT:f↦Λθ↦ reconstruction uT which corresponds to a solution of a model-based reconstruction (here shown for TV denoising) with a neural network-inferred spatially varying regularisation parameter Λθ .
Figure 7: TV denoising with spatially varying regularisation weights Λ of different regularity. Note that in the discrete setting, weights of high regularity are understood as the discrete analogues of continuous weights, with smooth transitions between the weight values; see [ 31 ] . On the other hand, weights of low regularity are understood as discrete analogues of discontinuous weights, with abrupt transitions. The latter ones can better adapt to the image details, leading to better reconstructions as the images above showcase. Note the different scale between the two weights.
Figure 8: Denoising: Applying Λ -maps to different instances of the noise. An experiment that shows that the parameter maps Λ produced by the neural network-based unrolled approach of Fig. 6 (scheme ( 27 )–( 28 )) are highly adapted not only to the image content but also to the specific noise instance. Here, two images with different noise instances of the same zero-mean Gaussian distribution ( σ2=0.04 ) were generated and used to infer the spatially adapted maps Λ1 and Λ2 for TV denoising. When these maps are cross-applied to the two noisy images, they yield considerably worse results.
Figure 9: MRI reconstruction: Applying Λ -maps to data created with different instances of the noise under the same undersampling mask. Series of MRI reconstructions under a common sampling mask but with two different noise instances of the same Gaussian distribution added to the k -space data. Spatially varying regularisation maps are produced by the neural network-based unrolled approach of Fig. 6 (scheme ( 27 )–( 28 )). Unlike the denoising case of Fig. 8 , the cross application of the Λ -maps yields comparable reconstructions.
Figure 10: MRI reconstruction: Applying Λ -maps to data created with different undersampling masks under the same instance of the noise. Series of MRI reconstructions under a common noise instance but with two different undersampling masks of the k -space data of the same rate. Spatially varying regularisation maps are produced by the neural network-based unrolled approach of Fig. 6 (scheme ( 27 )–( 28 )). Unlike the denoising case of Fig. 8 , the cross application of the Λ -maps yields comparable reconstructions as in the case of Fig. 9 .
Figure 11: MRI reconstruction: Applying Λ -maps to data created with different undersampling masks under two different instances of the noise. Series of MRI reconstructions under two different noise instances of the same Gaussian distribution and with two different undersampling masks of the k -space data of the same rate. Spatially varying regularisation maps are produced by the neural network-based unrolled approach of Fig. 6 (scheme ( 27 )–( 28 )). Unlike the denoising case of Fig. 8 , the cross application of the Λ -maps yields comparable reconstructions as in the case of Fig. 9 and Fig. 10 .
Faculty of Mathematics, Chemnitz University of Technology, Germany · NeuroSpin, Frédéric Joliot Institute for Life Sciences, CEA Paris-Saclay, France · MIND, Inria Saclay Centre, France +1
Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China · School of Mathematical Sciences, Nankai University, Tianjin, 300071, China · School of Mathematical Sciences and Institute of Mathematics and Interdisciplinary Sciences, Tianjin Normal University, Tianjin, 300387, China