Diffusion-based models enable monocular geometry estimation, yet their pixel-space precision is limited by a shared, under-studied error source: VAE reconstruction degradation. The 8x spatial compression in the VAE encoder-decoder degrades surface normals at object boundaries; even encoding and decoding ground-truth normals introduces 1.3--8.5° of mean angular error (MAE), with edge MAE reaching 2.8x the global MAE. We present TransNormal-2, a FLUX.2-based rectified-flow framework with single-step deterministic inference that addresses this degradation on both sides of the VAE decoder: in how latent predictions are supervised during training, and in how decoded normals are corrected at inference. First, geometry-aware pixel-space losses, including inverse rendering self-consistency, von~Mises-Fisher angular loss, and wavelet edge-aware regularization, complement latent MSE by enforcing spherical normal geometry and diffuse image-formation cues after VAE decoding. Second, a lightweight Geometric Refinement Module (GRM) applies an RGB-guided residual correction to reduce boundary-localized decoding errors without freely rewriting the coarse prediction. On general-scene benchmarks, TransNormal-2 matches or exceeds MoGe-2 on all eight reported metrics while using only 1.4% as many task-specific normal annotations. The gains are clearest for transparent objects, reducing MAE by 4.2° on ClearGrasp and 3.1° on ClearPose over the strongest prior baselines. Code will be released at https://longxiang-ai.github.io/TransNormal-2.
Figures & tables
Fig. 1 : In-the-wild qualitative results on surface normal estimation. TransNormal-2 (ours) recovers accurate surface normals on transparent objects. Scene 1 : prism under extreme lighting. Scene 2 : double-walled glass. Scene 3 : balls viewed through a glass plate. (§ V-B )
Fig. 2 : VAE reconstruction degradation. (a) Mean angular error (MAE, ∘ ) from encoding and decoding ground-truth normals with the frozen VAE; the Ratio column reports Edge/Global MAE. (b) The encode → decode reconstruction and its full-frame angular error map (bright = high error) show that errors concentrate at geometric boundaries, consistent with boundary detail being blurred by the 8× latent bottleneck. (§ III-A )
Fig. 3 : Overview of the TransNormal-2 framework. (a) Encoding : the frozen FLUX.2 VAE encoder Evae extracts and patchifies the RGB latent zrgb into image tokens himg ; (b) Single-Step Direct Prediction : the fine-tuned FLUX.2[klein] DiT fθ directly predicts the normal latent z^n from zrgb at a fixed timestep t under empty-prompt conditioning, processing the tokens through the LoRA-adapted double-stream and single-stream blocks; (c) Decoding & Refinement : the DiT output passes through a Local Continuity Module (LCM) in latent space, then the frozen FLUX.2 VAE decoder Dvae reconstructs the coarse geometry map, which is further refined by the Geometric Refinement Module (GRM) using the input RGB’s edge information. Training combines latent normal MSE ( Lnormal ), wavelet edge-aware regularization ( Lwavelet ), von Mises-Fisher angular loss ( Lvmf ), and inverse rendering self-consistency loss ( Lrender ). (§ IV )
Fig. 4 : Geometry-aware pixel-space losses. (Left) The vMF loss acts on decoded unit normals on S2 : maximizing n^⊤n∗=cosθ shrinks the geodesic angle θ between prediction and ground truth, a quantity that the latent-space MSE on VAE codes never sees. The Euclidean chord is shown for reference; for unit vectors it is a monotone function of θ . (Right) Inverse rendering self-consistency estimates a gradient-detached lighting vector ℓ^det , renders Lambertian shading from N^ , and minimizes Lrender=1−NCC(S^,Igray) , with panel (d) showing standardized local NCC evidence. Together, the losses enforce spherical normal geometry and diffuse image formation in pixel space. (§ IV-C )
Method
Venue
Data ↓
NYUv2
ScanNet
iBims
Sintel
Avg.
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Rank
Omnidata V2 † [ 84 ]
CVPR 22
12.2M
17.2
55.5
16.2
60.2
18.2
63.9
40.5
14.7
12.7
DSINE [ 17 ]
CVPR 24
160K
16.4
59.6
16.2
61.0
17.1
67.4
34.9
21.5
8.9
GeoWizard ⋆ [ 23 ]
ECCV 24
280K
18.9
50.7
17.4
53.8
19.3
63.0
40.3
12.3
14.5
StableNormal ⋆ [ 24 ]
SA 24
250K
18.6
53.5
17.1
57.4
18.2
65.0
36.7
14.1
13.0
E2E-FT ⋆ [ 4 ]
WACV 25
74K
16.5
60.4
14.7
66.1
16.1
69.7
33.5
22.3
6.0
TABLE I : Quantitative comparison on general scene normal estimation . We evaluate on NYUv2, ScanNet, iBims, and Sintel datasets. Metrics: mean angular error (Mean ↓ ) and percentage of pixels within 11.25∘ ( ↑ ). TransNormal-2 matches or exceeds MoGe-2 on all eight general-scene metrics while using only 122K training samples ( 1.4% of MoGe-2’s 8.9M). The best , second best , and third best results are highlighted. ⋆ : diffusion-based; † : transformer-based; ‡ : trained on Sintel (not zero-shot). SA: SIGGRAPH Asia. (§ V-C )
Method
Venue
Data ↓
ClearGrasp (Synthetic)
TN-Syn
ClearPose (Real-World)
Avg.
Mean ↓
11.25∘↑
30∘↑
Mean ↓
11.25∘↑
30∘↑
Mean ↓
11.25∘↑
30∘↑
Rank
Omnidata [ 16 ]
ICCV 21
12.2M
36.9
15.1
49.1
11.3
80.9
89.3
48.3
10.8
33.8
17.1
Omnidata V2 † [ 84 ]
CVPR 22
12.2M
33.8
18.3
55.9
8.2
87.0
92.6
51.7
13.8
33.2
15.6
DSINE [ 17 ]
CVPR 24
160K
25.7
26.4
68.6
13.2
70.3
90.7
40.2
15.9
46.3
14.2
Marigold ⋆ [ 2 ]
CVPR 24
74K
27.6
31.0
65.3
6.2
90.4
96.3
33.0
25.5
57.5
10.8
GeoWizard ⋆ [ 23 ]
ECCV 24
280K
31.3
20.8
59.5
9.4
78.9
95.0
36.8
14.2
49.7
14.8
TABLE II : Quantitative comparison on transparent object normal estimation . We evaluate on ClearGrasp, our proposed TN-Syn, and ClearPose datasets. Metrics: mean angular error (Mean ↓ , lower is better) and percentage of pixels within 11.25∘ and 30∘ thresholds ( ↑ , higher is better). Data ↓ : number of labeled training samples for the geometric task. TransNormal-2 achieves the best result on every reported metric using only 122K samples; gains are largest on ClearGrasp and ClearPose, while TN-Syn is near-saturated. See Table I for general scene results. The best , second best , and third best results are highlighted. ⋆ : diffusion-based; † : transformer-based; ‡ : trained on Sintel (not zero-shot). SA: SIGGRAPH Asia. (§ V-C )
Fig. 5 : Qualitative results on unlabeled in-the-wild general objects. We compare TransNormal-2 with representative baselines on examples from in-the-wild images. TransNormal-2 produces coherent object-scale geometry while preserving fine structures on curved ceramic surfaces and mechanical parts. Please zoom in for details. (§ V-B )
Fig. 6 : Qualitative comparison on transparent object normal estimation. We compare our method against state-of-the-art approaches across TN-Syn, ClearGrasp, and ClearPose datasets. For each dataset, the top row shows predicted normals and the bottom row shows angular error maps (blue: low, red: high). Even on ClearPose, an extremely challenging real-world dataset with diverse transparent objects under cluttered scenes, our method produces lower-error zero-shot predictions than the compared approaches. Existing methods often produce blurry or incorrect normals on transparent regions due to refraction, while our method preserves more coherent surface geometry. Please zoom in for details. (§ V-B )
Component
TransNormal
TransNormal-2
Backbone
SD 2.0 (U-Net, 865M)
FLUX.2[klein] (DiT, 9B)
Adaptation
Full fine-tune
LoRA ( r=256 )
Inference
Single-step prediction
Single-step prediction
Training loss
MSE + Wavelet
MSE + Wavelet + vMF + Rendering
Decode fix
None
GRM ( ∼ 0.4M params)
Training data
CG + TN-Syn + HS + VK
CG + TN-Syn + HS + VK
TABLE III : TransNormal vs. TransNormal-2: Key differences and head-to-head mean angular error ( ∘ , ↓ ). TransNormal denotes our prior model [ 1 ] ; both methods are evaluated under identical protocols. (§ V-C )
Configuration
NYUv2
ScanNet
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Group A: Training recipe (cumulative)
(a)
MSE only
16.9
58.8
14.6
64.9
(b)
+ Wavelet + vMF
16.6
58.5
14.3
65.8
(c)
+ Rendering loss
16.0
61.1
13.7
68.2
Group B: Decode refinement (from row c)
TABLE IV : Ablation on key design choices. Group A: cumulative training-recipe ablation; rows (a)–(b) are 20K-step from-scratch controls, and row (c) uses the full staged Phase-1 schedule. Group B: decode refinement. Row (f) is full TransNormal-2. (§ V-D )
Fig. 7 : Qualitative ablation on in-the-wild objects. Starting from the MSE-only baseline, wavelet and vMF supervision recover more coherent local geometry, inverse-rendering self-consistency further improves structure, and the full model with the GRM reduces residual boundary artifacts. Panel letters (b)–(e) correspond to rows (a), (b), (c), and (f) of Table IV , respectively. (§ V-D )
Configuration
#Params
NYUv2
ScanNet
iBims
Sintel
CG-Syn
TN-Syn
ClearPose
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
K=2 , hidden = 64
234K
14.7
62.5
12.7
69.2
14.8
70.6
29.2
27.6
11.2
65.1
3.6
95.8
19.1
52.3
K=8 , hidden = 64
716K
14.7
62.5
12.7
69.1
14.8
70.5
29.2
27.4
11.3
65.0
3.6
95.8
19.1
52.3
K=4 , hidden = 128
1.47M
14.8
62.3
12.8
68.9
14.7
70.9
29.1
27.7
11.3
64.9
3.5
95.8
19.1
52.2
K=4 , hidden = 64, w/o RGB
356K
15.7
61.5
13.4
68.6
14.6
72.1
28.8
28.8
11.2
65.2
3.5
95.8
19.1
52.3
K=4 , hidden = 64 (default)
∼ 0.4M
14.7
62.5
12.7
69.2
14.7
70.6
29.2
27.5
11.3
65.0
3.6
95.8
19.1
52.3
TABLE V : GRM design ablation. We ablate the depth ( K ), width (hidden channels), and input design of the Geometric Refinement Module while keeping the final core predictor, GRM training losses, two-stage GRM training protocol, and inference settings fixed. Metrics: mean angular error (Mean ↓ ) and percentage of pixels within 11.25∘ ( ↑ ). The RGB-guided depth and width variants are highly stable, with average mean angular error varying by only 0.03∘ across K=2 , K=4 , K=8 , and hidden = 128. We use K=4 , hidden = 64 as a conservative accuracy-parameter-stability tradeoff. Removing RGB guidance mainly hurts NYUv2 and ScanNet, while remaining competitive on synthetic and transparent benchmarks. (§ V-E )
Rank
#Params
General
Transparent
Avg ↓
NYUv2
ScanNet
iBims
Sintel
CG-Syn
TN-Syn
ClearPose
64
∼ 175M
16.6
14.6
15.9
30.6
12.0
4.6
21.1
16.5
128
∼ 350M
16.0
14.1
15.7
30.6
12.3
5.4
21.0
16.5
256
∼ 700M
16.0
13.7
14.8
28.9
11.4
3.9
19.3
15.4
512
∼ 1.4B
16.8
14.6
16.0
31.2
12.6
4.0
20.6
16.5
TABLE VI : LoRA rank ablation. We vary the LoRA rank r (with α=r ) and evaluate the core predictor with GRM refinement disabled, isolating the capacity of the DiT LoRA adapter. Rank 256 gives the best average error and matches or leads the best rounded value on every dataset, with no further gains at rank 512. (§ V-F )
Method
NYUv2
ScanNet
CG-Syn
TN-Syn
Avg.
No refinement
16.0
13.7
11.4
3.9
11.3
Guided Filter ( rGF=4 )
14.8
12.8
12.4
4.7
11.2
Guided Filter ( rGF=8 )
15.0
12.9
14.5
5.6
12.0
Guided Filter ( rGF=16 )
16.3
13.8
18.8
7.1
14.0
Joint Bilateral
15.0
13.0
11.7
4.2
11.0
Bilateral
15.4
13.4
11.4
3.9
11.0
TABLE VII : Learned GRM vs. classical image-guided filters. All rows refine the same coarse normal predictions and are evaluated with the harness of the main tables, so “No refinement” matches the core-predictor results reported there. Mean angular error ( ∘ , ↓ ); the GRM is best on every dataset and on average. (§ V-G )
Edge ( ∘ )
Non-Edge ( ∘ )
Δ ( ∘ )
Dataset
w/o
w/
w/o
w/
Edge
Non-E
NYUv2
26.1
23.5
13.7
12.9
−2.6
−0.8
ScanNet
26.0
23.4
12.5
11.9
−2.6
−0.6
TABLE VIII : Edge vs. non-edge MAE analysis. Pixels are separated into edge (Canny on RGB, dilated 3×3 ) and non-edge regions. We compare the final core predictor with and without the GRM on general-scene benchmarks, where RGB edges provide a reliable proxy for geometric discontinuities. Negative Δ means lower MAE after GRM. (§ V-H )
Fig. 8 : Frequency-wise ratio of angular-error PSD after and before GRM refinement. Values below 1 indicate lower spatial error energy after refinement. The shaded region marks the highest third of radial-frequency bins; integrated energy in this band decreases by 53.2% on NYUv2 and 42.9% on ScanNet. (§ V-I )
Surface Normal Estimation recovers the orientation of visible 3D surfaces from images and other sensor data for shape understanding and reconstruction.
Stereo matching and surface normal estimation are fundamental tasks in 3D vision. However, existing feed-forward stereo methods still struggle to produce reliable predictions in challenging regions, mainly due to the lack of strong geometric priors. In this paper, we propose GeoStereo, a unified stereo geometry estimation framework that leverages powerful diffusion priors to jointly predict disparity and surface normals. Specifically, GeoStereo couples a feed-forward stereo matching pipeline with a diffusion-based normal estimation branch. To enable effective interaction between the two tasks, we introduce a disparity to normal initialization strategy and construct a warp to left-view condition for the diffusion process. This coupled design allows the diffusion branch to provide strong structural priors that enhance disparity estimation in ill-posed regions, while the feed-forward branch offers reliable geometric guidance for accurate normal prediction. Extensive experiments show that GeoStereo performs reliably in challenging scenarios, including low-light environments, highly reflective surfaces, and transparent objects. Under zero-shot settings, it achieves Rank-1 disparity estimation on multiple benchmarks, including KITTI and NYUv2, and delivers the best normal estimation accuracy on many real indoor benchmarks, such as iBims-1 and ScanNet. Project page: https://qz-wei.github.io/GeoStereo.github.io/
Qizhe Wei, Xianda Guo, Shaocong Xu +3
Beijing Institute of Technology BAAI Beijing, China · School of Computer Science, Wuhan University Wuhan, China · BAAI Beijing, China +3
This work presents ViGeo, a feed-forward foundation model for recovering spatially dense and temporally consistent geometry from video sequences. Built upon a plain transformer architecture without task-specific architectural modifications, ViGeo supports streaming, full-sequence, and long-video inference within a unified model. The key design is dynamic chunking attention, which exposes the model to both bidirectional and causal temporal contexts during training and allows it to adapt its attention pattern at test time without retraining. To improve supervision quality, we further introduce a completion-based data refinement framework. This framework trains a video depth completion teacher that conditions on sparse and noisy annotations and exploits video/multi-view context to produce dense, temporally coherent, and geometrically reliable training targets. Beyond depth and point maps, ViGeo also predicts surface normals within the same framework. Trained solely on public datasets, ViGeo achieves state-of-the-art performance across online, offline, and long-video depth estimation, surface normal estimation, and video point map estimation.
Zhu Yu, Jingnan Gao, Runmin Zhang +9
Zhejiang University · Shanghai Jiao Tong University · Tongyi Lab, Alibaba Group +1
Joint estimation of surface normals and depth is essential for holistic 3D scene understanding, yet high-resolution prediction remains difficult due to the trade-off between preserving fine local detail and maintaining global consistency. To address this challenge, we propose the Ultra Resolution Geometry Transformer (URGT), which adapts the Visual Geometry Grounded Transformer (VGGT) into a unified multi-patch transformer for monocular high-resolution depth--normal estimation. A single high-resolution image is partitioned into patches that are augmented with coarse depth and normal priors from pre-trained models, and jointly processed in a single forward pass to predict refined geometric outputs. Global coherence is enforced through cross-patch attention, which enables long-range geometric reasoning and seamless propagation of information across patches within a shared backbone. To further enhance spatial robustness, we introduce a GridMix patch sampling strategy that probabilistically samples grid configurations during training, improving inter-patch consistency and generalization. Our method achieves state-of-the-art results on UnrealStereo4K, jointly improving depth and normal estimation, reducing AbsRel from 0.0582 to 0.0291, RMSE from 2.17 to 1.31, and lowering mean angular error from 23.36 degrees to 18.51 degrees, while producing sharper and more stable geometry. The proposed multi-patch framework also demonstrates strong zero-shot and cross-domain generalization and scales effectively to very high resolutions, offering an efficient and extensible solution for high-quality geometry refinement.