LFX: Fusion-Form-Flexible Light Field Modeling for Dense Semantic Segmentation and Salient Object Detection
Authors: Fei Teng, Lingxin Huang, Buyin Deng, Kai Luo, Boyuan Zheng, Zheng Fang, Hong Zheng, Kunyu Peng, Jiaming Zhang, Yaonan Wang, Kailun Yang
Organizations: The authors are with the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · The authors are with China Mobile Group Hunan Company Ltd., China. · The author is with the Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany.
Two distinct light field (LF) observation forms, namely sub-aperture images (S.Aperture images) and focal stack (FS), correspond to two fusion settings: the central image with S.Aperture images (CS) and the central image with FS (CF). Existing LF perception methods are typically form-specific, while a modeling design that can be separately instantiated under either setting remains underexplored. To address this gap, we present LFX, a form-flexible LF perception framework. Here, ``X'' represents the LF observation form, indicating that the one modeling design can be separately instantiated under two fusion settings (CS and CF). Specifically, we propose Field-of-Parallax Angular Subspace Modeling (FoP-ASM), which maps the feature difference between each auxiliary observation and the central image to an observation-specific angular marker. This avoids pairwise modeling among auxiliary observations and form-specific geometric assumptions. Meanwhile, a shared manifold subspace and the associated regularization losses constrain the angular residuals within a common feature space, promoting semantic consistency across auxiliary observations. Extensive evaluations across three LF datasets show that LFX achieves state-of-the-art results across distinct LF forms. Under the S.Aperture images and FS settings, respectively, LFX reduces MAE by 12.12%/20.59% on DUTLF-V2 and by 6.15%/7.69% on HFUT, while achieving the best performance on UrbanLF-real. The source code will be made publicly available at https://github.com/FeiT-FeiTeng/LFX.
Figures & tables
Fig. 1: Comparison of LF observation forms and modeling settings. (a) S.Aperture images introduce viewpoint-induced disparity shifts, whereas FS contains refocused observations at different focal planes. (b) Across different LF forms, comparing each auxiliary observation with the central image reveals three shared properties: contextual consistency, global response difference, and local response difference. (c) Existing form-specific methods model auxiliary-to-auxiliary disparity or focus relationships, whereas LFX can be instantiated and trained separately under either CS or CF.
Fig. 2: Overall framework of LFX. (a) illustrates the encoding stages of LFX, while (b) presents the architecture of the proposed FoP-ASM module. By independently modeling each auxiliary image and enforcing contextual consistency, our framework supports heterogeneous LF forms and achieves state-of-the-art performance on both salient object detection and semantic segmentation tasks.
Method
MAE ↓
E ↑
S ↑
F ↑
Avg.R ↓
2D Methods
BBRF [ 58 ]
0.051
0.907
0.869
0.843
–
LeNo [ 59 ]
0.047
0.922
0.869
0.868
–
ConTriNet [ 60 ]
0.057
0.903
0.847
0.806
–
MDSAM [ 61 ]
0.044
0.940
0.940
0.898
–
S.Aperture images
TABLE I: Results on DUTLF-V2 [ 24 ] under S.Aperture images and FS forms. “ ↑ ” indicates that a higher value is preferred, “ ↓ ” indicates that a lower value is preferred, and “A vg.R ” indicates the average ranking. Avg.R and Gain are computed among LF methods within each form, excluding the 2D methods.
Method
MAE ↓
E ↑
S ↑
F ↑
Avg.R ↓
S.Aperture images
DLFS [ 66 ]
0.109
0.755
0.712
0.592
8.88
LFDCN [ 25 ]
0.109
0.797
0.730
0.671
7.75
OBGNet [ 63 ]
0.094
0.790
0.740
0.671
7.38
MTCNet [ 32 ]
0.084
0.818
0.751
0.714
5.13
GAGNN [ 74 ]
0.069
0.844
0.798
0.762
3.25
TABLE II: Quantitative comparison on the HFUT [ 25 ] dataset under S.Aperture images and FS forms. “ ↑ ” indicates that a higher value is better, whereas “ ↓ ” indicates that a lower value is better, and “A vg.R ” indicates the average ranking.
Model
Acc(%) ↑
mAcc(%) ↑
mIoU(%) ↑
Avg.R
S.Aperture images
Sigma [ 73 ]
87.94
75.67
68.09
11.00
PSPNet-LF [ 80 ]
91.75
84.31
77.79
10.00
OCR-LF [ 81 ]
92.92
86.50
80.69
6.67
LF-IENet 5 [ 35 ]
92.40
84.94
79.03
9.00
LF-IENet 4 [ 35 ]
93.02
86.69
80.35
6.33
TABLE III: Semantic segmentation results on UrbanLF-real under S.Aperture images and FS forms. “↑” indicates that a higher value is preferred, and “A vg.R ” indicates the average ranking.
Fig. 3: Qualitative visualization of difference maps for the SOD task in the DUTLF-V2 [ 24 ] dataset. In each difference map, red regions denote prediction errors, while white and black regions indicate correctly predicted regions. The gray region denotes the original image and ground truth, the blue region denotes visualization results of methods using S.Aperture images as input, and the pink region denotes visualization results of methods using FS images as input.
Fig. 4: Qualitative comparison on the UrbanLF-real dataset [ 26 ] for the semantic segmentation task. Compared with existing methods, our method produces segmentation results with clearer boundaries and better semantic consistency.
Fig. 5: Challenging examples for LF perception. While our method demonstrates superior performance in capturing fine-grained structures compared to existing approaches, perception under complex lighting and environmental conditions remains highly challenging for all methods.
Method
Angular Marker
Mani.
S.Reg.
FRM
FFM
mIoU ↑
mAcc ↑
LFX
✓
✓
✓
✓
✓
76.45
83.49
Exp.1
×
✓
✓
✓
✓
75.21(-1.24)
84.18(+0.69)
Exp.2
×
✓
×
✓
✓
75.02(-1.43)
83.36(-0.13)
Exp.3
×
×
×
✓
✓
74.13(-2.32)
82.71(-0.78)
Exp.4
×
×
×
×
✓
73.34(-3.11)
82.37(-1.12)
Exp.5
×
×
×
×
×
71.89(-4.56)
81.27(-2.22)
TABLE IV: Ablation study for the core components of LFX. “M ani. ” denotes the manifold subspace modeling module, and “ S.Reg. ” denotes the subspace regularization term. “FRM” and “FFM” denote the Feature Rectification Module and Feature Fusion Module [ 2 , 52 ] , respectively. The ‘↑” indicates that a higher value is preferred.
Views
Segmentation
Salient Detection (FS)
mIoU ↑
mAcc ↑
GFLOPs
MAE (W.O) ↓
MAE (W.) ↓
GFLOPs
Aux.2
75.07
83.92
63.06
0.037
0.034
15.05
Aux.5
76.45(+1.38)
83.49(-0.43)
67.59
0.039(+0.002)
0.032(-0.002)
16.13
Aux.9
76.03(+0.96)
83.82(-0.10)
73.62
0.035(-0.002)
0.032(-0.002)
17.57
Aux.33*
77.34(+2.27)
85.50(+1.58)
109.82
0.036(-0.001)
0.030(-0.004)
18.65
TABLE V: Ablation on auxiliary observations. “A ux . 33*” indicates that the auxiliary observations are set to 33 for S.Aperture images and 12 for focal stacks. The ‘↑” indicates that a higher value is preferred, and ↓” denotes that a lower value is better for the given evaluation metric. “W.O” denotes without FoP-ASM, and “W.” denotes with FoP-ASM. “GFLOPs” denotes the computational cost. For each metric column, the parenthetical value reports the signed change relative to the corresponding result under Aux.2.
Fig. 6: t-SNE visualization of central image features before and after the FoP-ASM module on the DUTLF-V2 dataset [ 24 ] . Each row corresponds to one image sample. To analyze feature organization under different semantic granularities, the t-SNE clustering is performed with 2, 3, and 5 categories. For each setting, we compare two representative LF forms: S.Aperture images (disparity-centric form) and focal stack (refocus-centric form) .
TABLE VI: Stage-wise ablation for FoP-ASM. We report the performance, number of parameters, and computational overhead when progressively removing FoP-ASM from different encoder stages. The symbols “ ✓ ” and “ × ” indicate whether FoP-ASM is applied to each encoder stage, ranging from Stage 1 to Stage 4. Those experiments progressively remove FoP-ASM from later encoder stages. “P arams .” is the abbreviation for parameters. “E xp .” is the abbreviation for experiment.
Method
Differ. Map
Ch.P
So.P
ADD.
mIoU ↑
mAcc ↑
LFX
✓
✓
✓
Cat.
76.45
83.49
Exp.1
✓
✓
✓
✓
76.10(-0.35)
83.71(+0.22)
Exp.2
✓
×
✓
×
75.50(-0.95)
83.02(-0.47)
Exp.3
✓
✓
×
×
75.79(-0.66)
82.74(-0.75)
Exp.4
×
×
×
×
75.21(-1.24)
84.18(+0.69)
TABLE VII: Ablation study on the construction of the angular marker. “Differ.Map” denotes the difference map between each auxiliary observation and the central image. “Ch.P” indicates Ps , while “So.P” denotes As . “Add.” and “Cat.” denote element-wise addition and concatenation, respectively.
Fig. 7: Heatmap visualization of global response difference and local response difference for S.Aperture images and FS.
Method
Acc ↑
mAcc ↑
mIoU ↑
Avg.R ↓
Params. ↓
GFLOPs ↓
SHIFNet 5 [ 82 ]
91.31
84.51
76.01
4.33
78.47M
416.79
SACNet 5 [ 83 ]
91.94
85.58
77.93
2.33
327.66M
596.85
CMNeXt 5 [ 2 ]
91.99
84.83
77.16
2.67
116.61M
128.51
StitchFusion 5 [ 79 ]
91.49
83.37
75.32
4.67
69.98M
310.41
LFX 5 (Ours)
92.65
86.46
79.89
1.00
116.86M
129.59
TABLE VIII: Performance and efficiency comparison on UrbanLF-real under the FS form. “A vg.R ” denotes the average ranking.
Department of EE, IIT Kharagpur, India · Rekhi Centre of Excellence for the Science of Happiness, IIT Kharagpur, India · Department of E&ECE, IIT Kharagpur, India
Capital Normal University Information Engineering College, Beijing 100048, China · Bigo Technology Pte. Ltd., Beijing 100020, China · Explorer Global (Suzhou) Artificial Intelligence Technology Co., Ltd., Suzhou, Jiangsu 215123, China +1
Department of Artificial Intelligence (AI), Sungkyunkwan University (SKKU), Suwon, South Korea · Department of Electrical and Computer Engineering, SKKU, Suwon, South Korea