Organizations: School of Cyber Science and Engineering, Southeast University, Nanjing 210096, China. · Department of Electronic and Computer Engineering, the Hong Kong University of Science and Technology, Hong Kong, China. · College of Electronic and Information Engineering, Southwest University, Chongqing, China. · Institute of AI for Industries, Chinese Academy of Sciences, Nanjing, China. · Surrey Institute for People-Centred Artificial Intelligence and the Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, U.K.
Generating consistent multi-view RGB images and normal maps from a single input image provides complementary appearance and geometric cues for 3D reconstruction. However, maintaining cross-view consistency requires information exchange across viewpoints, and dense cross-view attention becomes computationally expensive as the number of views and spatial resolution increase. In this paper, we propose Global-Local Proxy Attention (GLPA) to reduce cross-view communication costs while preserving fine-grained within-view processing. Specifically, GLPA retains full-resolution interactions within each view and uses compact global proxies for cross-view communication. To further leverage the exchanged information, GLPA incorporates iterative proxy refinement, which rebuilds the proxies from updated features and reuses the same attention parameters for subsequent interactions within a single denoising step. We evaluate GLPA on two structurally different diffusion backbones, FLUX and Stable Diffusion. Experiments on FLUX show that GLPA achieves a 2.65× speedup in attention computation per denoising step over dense attention, while improving reconstruction IoU from 0.6849 to 0.7019 and reducing NVS LPIPS from 0.086 to 0.076. Results on Stable Diffusion further demonstrate the effectiveness of GLPA, with a 1.80× attention speedup and improvements across all five quality metrics.
Figures & tables
Fig. 1: Consistent multi-view generation with GLPA. By combining global-local proxy interaction with iterative proxy refinement, GLPA improves cross-view consistency compared with reduced attention while requiring less attention computation than dense attention.
Fig. 2: Overall pipeline incorporating GLPA. GLPA combines global-local proxy interaction with iterative proxy refinement to preserve full-resolution within-view interactions and enable efficient cross-view communication. With GLPA integrated into multi-view diffusion, the pipeline generates consistent multi-view RGB images and normal maps from a single input image for downstream textured 3D reconstruction.
Fig. 3: Visualization of a single round of global-local proxy interaction in GLPA. Each row shows a query cell on the front view (red box), the within-view attention at this location, and the global attention of this cell over the proxies of three other views, displayed at their associated spatial locations. The proxy pathway reaches the corresponding parts across viewpoints, while the local pathway preserves fine within-view structure.
Fig. 4: Qualitative comparison with baseline methods on novel view synthesis.
Methods
CD ↓
IoU ↑
LPIPS ↓
SSIM ↑
PSNR ↑
RealFusion [ 46 ]
0.0819
0.2714
0.283
0.722
15.26
Zero-1-to-3 [ 6 ]
0.0339
0.5035
0.166
0.779
18.93
One-2-3-45 [ 47 ]
0.0629
0.4086
-
-
-
Shap-E [ 48 ]
0.0436
0.3584
-
-
-
Magic123 [ 49 ]
0.0516
0.4528
-
-
-
SyncDreamer [ 8 ]
0.0261
0.5421
0.146
0.798
20.05
TABLE I: Quantitative Comparison on the GSO Benchmark Using Chamfer Distance (CD) and IoU for 3D Reconstruction, and LPIPS, SSIM, and PSNR for Novel View Synthesis.
Fig. 5: Qualitative comparison of 3D reconstruction results on the GSO dataset. The GLPA-based pipeline generates the best results with more views and high resolution, compared with existing methods.
Fig. 6: Qualitative comparison of 3D reconstructions on in-the-wild data. Compared with previous methods, the GLPA-based pipeline produces more complete and higher-quality 3D models.
Methods
N-view
Resolution
Attention runtime/step (s)
CD ↓
IoU ↑
LPIPS ↓
SSIM ↑
PSNR ↑
FLUX.1-dev Backbone
Dense
8
512
1.637
0.0193
0.6849
0.086
0.922
28.03
GLPA (w/o refinement)
8
512
0.390
0.0246
0.6156
0.101
0.914
26.75
GLPA (reuse proxy)
8
512
0.598
0.0254
0.6246
0.087
0.920
27.44
GLPA
8
512
0.617
0.0185
0.7019
0.076
0.927
27.80
SD2.1-UnCLIP Backbone
TABLE II: Attention Runtime and Quality Comparison of Dense Attention and GLPA Variants on FLUX and SD Backbones.
s
R
Attention runtime/step (s)
CD ↓
IoU ↑
LPIPS ↓
SSIM ↑
PSNR ↑
2
1
0.390
0.0246
0.6156
0.101
0.914
26.75
2
2
0.617
0.0185
0.7019
0.076
0.927
27.80
2
3
0.799
0.0262
0.6210
0.084
0.921
27.41
4
2
0.370
0.0308
0.5578
0.096
0.917
27.47
8
2
0.324
0.0404
0.4496
0.125
0.908
25.75
TABLE III: Effect of Pooling Stride and Interaction Rounds on Attention Efficiency, Novel View Synthesis, and 3D Reconstruction.
Methods
Latency (ms)
Peak memory (G)
LPIPS ↓
SSIM ↑
PSNR ↑
Dense
26.61
0.828
0.101
0.918
25.84
PT-DiT [ 38 ]
35.41
1.913
0.145
0.901
24.39
Agent Attention [ 36 ]
21.74
1.986
0.137
0.909
24.84
GLPA
22.07
0.601
0.111
0.910
25.86
TABLE IV: Efficiency and Quality Comparison of Alternative Attention Designs on the FLUX Backbone.
Methods
N-view
Resolution
Latency (ms)
Peak memory (G)
Dense
8
512
14.92
0.244
Era3D [ 10 ]
8
512
1.67
0.322
CTR3D [ 11 ]
8
512
9.20
0.325
GLPA
8
512
5.02
0.156
TABLE V: Efficiency Comparison of Attention Modules on the SD Backbone.
Rotary Positional Embedding (RoPE) is widely used to encode relative positions in Transformers. In multi-view settings, however, patch coordinates are defined independently in each image: the same 3D point can therefore have different coordinates across views, making them insufficient for establishing cross-view correspondences. We introduce WorldRoPE, a probabilistic world-centric RoPE for multi-view attention. At each layer, WorldRoPE back-projects each patch token into a set of volumetric Gaussian primitives in a shared world frame. These primitives represent multiple plausible 3D supports for a token, jointly capturing alternative depth hypotheses, depth uncertainty, and the finite spatial extent of its patch frustum. When a key token's support is projected into a query view, it induces a distribution over where that key may appear in the query image. WorldRoPE integrates the rotary operator over this distribution in closed form, combining rotations at the projected primitive centers according to their mixture weights with covariance-dependent attenuation. This makes attention less sensitive to uncertain key locations in the query view, without rasterizing or warping token features. The formulation reduces to deterministic RoPE when the support collapses to a point. Across layers, the predicted geometry guides cross-view attention, while the updated features refine the geometry used at the next layer. Experiments on novel view synthesis, depth estimation, and 3D geometry reconstruction demonstrate the effectiveness of WorldRoPE for multi-view Transformers.
Mijeong Kim, Kyujin Lim, Bohyung Han
ECE, Seoul National University · IPAI, Seoul National University
3D Gaussian Splatting (3DGS) has emerged as a prominent paradigm for 3D reconstruction and novel view synthesis. However, it remains vulnerable to severe artifacts when trained under sparse-view constraints. While recent methods attempt to rectify artifacts in rendered views using image diffusion models, they typically rely on multi-view self-attention to retrieve information from reference images. We observe that this mechanism often fails when the rendered novel views output by 3DGS are heavily corrupted: damaged query features lead to erroneous cross-view retrieval, resulting in inconsistent rendering refinement. To address this, we propose GeoQuery, a geometry-guided diffusion framework that integrates generative priors with explicit geometric cues via a novel Geometry-guided Cross-view Attention (GCA) mechanism. First, by leveraging predicted depth maps and camera poses, we construct a geometry-induced correspondence field to sample reference features, forming a geometry-aligned proxy query that replaces the corrupted rendering features. Furthermore, we design a new cross-view feature aggregation pipeline, in which we restrict the cross-view attention to a local window around each proxy query to effectively retrieve useful features while suppressing spurious matches. GeoQuery can be seamlessly integrated into existing diffusion-based pipelines, enabling robust reconstruction even under extreme view sparsity. Extensive experiments on sparse-view novel view synthesis and rendering artifact removal demonstrate the effectiveness of our approach.
Xiao Cao, Yuze Li, Youmin Zhang +4
University of Electronic Science and Technology of China & Rawmantic AI, China · Rawmantic AI, China · Tianjin University, China +1
Recent feed-forward 3D reconstruction methods, such as visual geometry transformers, have substantially advanced the traditional per-scene optimization paradigm by enabling effective multi-view reconstruction in a single forward pass. However, most existing methods struggle to achieve a balance between reconstruction quality and computational efficiency, which limits their scalability and efficiency. Although some efficient visual geometry transformers have recently emerged, they typically use the same sparsity ratio across layers and frames and lack mechanisms to adaptively learn representative tokens to capture global relationships, leading to suboptimal performance. In this work, we propose TurboVGGT, a novel approach that employs an efficient visual geometry transformer with adaptive alternating attention for fast multi-view 3D reconstruction. Specifically, TurboVGGT employs an end-to-end trainable framework that leverages adaptive sparse global attention guided by adaptive sparsity selection to capture global relationships across frames and uses frame attention to aggregate local details within each frame. In the adaptive sparse global attention, TurboVGGT adaptively learns representative tokens with varying sparsity levels for global geometry modeling, considering that token importance varies across frames, different attention layers operate on tokens at different levels of abstraction, and global dependencies rely on structurally informative regions. Extensive experiments on multiple 3D reconstruction benchmarks demonstrate that TurboVGGT achieves fast multi-view reconstruction while maintaining competitive reconstruction quality compared with state-of-the-art methods.
David Huang, Guile Wu, Chengjie Huang +2
Huawei Noah’s Ark Lab · University of Toronto · Foundation Model Department, Huawei