No Cones, No Duals: A Fast and Scalable Solver for NRSfM

Suryansh Kumar

cs.CVOct 25, 2026

No Cones, No Duals: A Fast and Scalable Solver for NRSfM

Authors: Suryansh Kumar

Organizations: Visual and Spatial AI Lab, VCCM Section, College of PVFA, Department of ECEN, Department of CSCE, Texas A&M University, College Station, Texas, USA

Abstract

Non-rigid structure from motion (NRSfM) aims to recover camera motion and time-varying 3D shape from tracked 2D image measurements. Recent high-performing methods show that shape-space factorization remains highly effective for this task. These methods often use costly solvers for two key steps. First, a semidefinite programming (SDP) cone solver to perform Euclidean correction. Next, an alternating direction method of multipliers (ADMM) solver to recover the low-rank shape. These steps often scale poorly to long sequences and dense point tracks. We propose a fast, scalable NRSfM solver that avoids these costly steps. For rotation estimation, we introduce a differentiable Euclidean correction that directly optimizes small factors for each sequence. A row-polar projection then maps each estimated camera block to the closest row-orthonormal matrix. For shape estimation, we introduce an unrolled spectral-proximal solver. It combines reprojection-gradient steps with weighted singular-value shrinkage in the reshuffled shape domain. This solver further uses FISTA-style restart and continuation. The spectral shrinkage operator itself is classical, but we embed it in a fixed differentiable solver, while preserving NRSfM's geometric constraints. The proposed approach is category-agnostic, requires no external training data, and maps naturally to modern accelerator-friendly linear algebra primitives. Across sparse, dense, noisy, missing-data, and long-sequence benchmarks, our solver achieves substantial speedups with competitive reconstruction accuracy, including 7x-1000x runtime improvements on dense sequences with 28,887 tracked points.

Figures & tables

Figure 1: Accuracy-runtime and dense-sequence behavior. (a) Accuracy-runtime Pareto comparison on CMU MoCap [ 5 ] . Smaller, faded markers show individual sequences, while larger markers show method averages (lower left is better). The proposed solver achieves the fastest runtime while retaining competitive reconstruction accuracy relative to recent shape-space baselines. (b) On dense sequences [ 18 ] with $28,887$ tracks, the proposed solver achieves large runtime gains over recent shape-space baselines such as TSM [ 57 ] and OPM [ 33 ] , while retaining competitive accuracy.

\Block [tikz=top color=orange!25]*-1 Stage	\Block [tikz=top color=lightgray!25]*-1 Current Solvers	\Block [tikz=top color=OliveGreen!25]*-1 Ours
Rotation	SDP/cone solver for Gram recovery	Factor-side $\mathbf{Y}_{k}$ + row-polar projection
Shape	ADMM with auxiliary/dual variables	Unrolled layer, WSVT shrinkage, restart
Camera	Orthographic camera constraints	row-polar camera projection
Computation	Cone, ADMM subproblem machinery	Accelerator-friendly solver

Table 1: No-cone, no-dual positioning relative to recent high-performing shape-space NRS f M factorization pipelines.

Data	$(F,P,K)$	BMM [ 15 ]	R-BMM [ 32 ]	OPM [ 33 ]	TSM [ 57 ] $(\mathbf{R}_{\mathrm{init}}/~\mathbf{R}_{\mathrm{ref}})$	Ours
Drink	(1102, 41) $K=11$	0.0086 (10.01)	0.0240 (108.91)	0.0096 (143.54)	0.0086/ 1.3950 (88.94)	0.0060 (8.51)
Pickup	(357, 41) $K=12$	0.1210 (09.68)	0.1271 (96.24)	0.1144 (98.87)	0.1210/ 1.3197 (14.79)	0.1082 (7.79)
Yoga	(307, 41) $K=11$	0.0945 (08.08)	0.0944 (70.08)	0.0972 (76.88)	0.0945/ 0.1114 (10.48)	0.0996 (7.73)
Stretch	(370, 41) $K=13$	0.0682 (08.45)	0.0682 (73.32)	0.0688 (115.06)	0.0682/ 1.4116 (19.40)	0.0647 (7.83)

Table 2:

e_{R}

and rotation-stage runtime comparison with baselines. All methods are re-simulated for given

K

. TSM uses two rotation estimates, i.e., BMM initialization

\mathbf{R}_{\mathrm{init}}

and the refined rotation

\mathbf{R}_{\mathrm{ref}}

after its refinement module . Our approach is considerably faster while achieving competitive or better rotation accuracy. The number in parentheses shows compute time in seconds.

Figure 2: Stable convergence of the proposed solver on CMU MoCap sequences. The data-fit values $f(\mathbf{S})$ for R-BMM [ 32 ] , AOW [ 24 ] , and OPM [ 33 ] are normalized for visualization. Qualitative 3D reconstruction results obtained using our solver are also shown for representative sequences. The 4 curves correspond to data-fit values on 4 standard CMU sequences.

Data	BMM [ 15 ]	R-BMM [ 32 ]	OPM [ 33 ]	TSM [ 57 ]	Ours
Drink	0.0266 (61.22)	0.0119 (38.75)	0.0071 (36.08)	0.0033 (142.96)	0.0166 (06.45)
Pickup	0.1731 (26.95)	0.0198 (11.44)	0.0152 (10.34)	0.0126 (12.31)	0.0267 (04.88)
Yoga	0.1150 (12.87)	0.0129 (07.60)	0.0122 (07.51)	0.0118 (09.62)	0.0148 (04.19)
Stretch	0.1034 (15.28)	0.0144 (09.32)	0.0124 (08.73)	0.0131 (14.24)	0.0128 (04.21)
Dance	0.1864 (68.68)	0.1499 (12.82)	0.1209 (8.98)	0.0921 (11.39)	0.1218 (6.39)
Face	0.0303 (18.68)	0.0179 (7.97)	0.0145 (6.140)	0.0144 (14.09)	0.0179 (4.66)

Table 3:

e_{3d}

and shape-stage runtime comparison with high-performance methods after rotations are fixed . Best

K

from the previous methods is used to compare

e_{3d}

and time. Our approach is faster and its

e_{3d}

compares favorably with competing methods. Numbers in parentheses denote computation time in seconds.

Figure 3: Qualitative results on NRS f M challenge dataset.

Data	CSF2 [ 22 ]	BMM [ 15 ]	R-BMM [ 32 ]	AOW [ 24 ]	BP [ 50 ]	OPM [ 33 ]	TSM [ 57 ]	Ours
Articulated	11.52	18.49	16.00	15.03	16.10	12.18	10.69	10.58
Balloon	10.14	10.39	7.84	8.05	8.29	6.29	7.28	5.86
Paper	9.72	8.94	10.69	10.45	6.70	8.86	7.91	10.42
Stretch	8.65	10.02	7.53	9.01	7.66	6.36	5.43	7.02
Tearing	12.04	14.23	16.34	16.20	11.26	10.91	10.77	10.61

Table 4: 3D reconstruction error comparison with state-of-the-art on [ 26 ] dataset.

Data	SMSR [ 7 ]	DS [ 16 ]	GM [ 37 ]	ND [ 58 ]	DV [ 18 ]	OPM [ 33 ]	TSM [ 57 ]	Ours
Seq.1	0.1893	0.0636	0.0443	-	0.0531	0.0624 (55.60)	0.2223 (10964.22)	0.0774 (8.29)
Seq.2	0.2133	0.0569	0.0381	-	0.0457	0.0451 (53.87)	0.1271 (10500.35)	0.0442 (8.18)
Seq.3	0.1345	0.0374	0.0294	0.0450 (3h+ train)	0.0346	0.0279 (100.72)	0.0393 (11624.89)	0.0382 (15.01)
Seq.4	0.0984	0.0428	0.0309	0.0490 (3h+ train)	0.0379	0.0419 (102.19)	0.0615 (11609.17)	0.0522 (15.16)

Table 5:

e_{3d}

and overall run-time comparison with high-performing approaches on dense sequences [ 18 ] . Timings for other methods are unavailable due to the lack of open-source code. Observation: Our solver is faster than baselines: OPM and TSM.

Panel of Figure 4 — Figure 4: (a) Overall compute time comparison with current SOTA approaches [ 57 , 33 ] on [ 26 ] dataset. Observation : Our approach compute time is significantly better as the number of frames and points increases (for e.g., Tearing seq. with $F=405$ , $P=132$ ). (b) Our approach compares favorably with other methods on missing dataset sequence.

Data	CSF2 [ 22 ]	BMM [ 15 ]	PND [ 42 ]	CNS [ 43 ]	R-BMM [ 32 ]	TSM [ 57 ]	Ours
Eating2	0.0923	0.0665	0.0701	0.0823	0.0633	0.0520	0.0651
Smoking1	0.1230	0.0844	0.1049	0.1073	0.0746	0.0738	0.0726
Directions	0.0787	0.0648	0.0536	0.0621	0.0595	0.0529	0.0643
Smoking	0.0702	0.0492	0.0622	0.0578	0.0363	0.0352	0.0337
Waiting2	0.1029	0.1081	0.1114	0.1142	0.1117	0.1035	0.1012

Table 6:

e_{3d}

on One-circle-type camera motion dataset [ 70 ] .

Figure 5: Importance of each loss term included in Eq.( 8 ). Variation in

e_{\mathbf{R}}

as we add the three loss terms.

Panel of Figure 6 — Figure 6: $e_{3d}$ variation w.r.t different noise levels in $\mathbf{W}$ .

Data	CSF1 [ 20 ]	CSF2 [ 22 ]	KSTA [ 21 ]	PTA [ 5 ]	NP [ 63 ]	PBezier [ 2 ]	PPTA [ 4 ]	Ours
Drink	0.0223	0.0223	0.0156	0.0250	0.0354	0.009	0.011	0.0166
Pickup	0.2301	0.2277	0.2322	0.2369	0.0284	0.138	0.235	0.0267
Yoga	0.1467	0.1464	0.1476	0.1624	0.0210	0.111	0.158	0.0148
Stretch	0.0710	0.0685	0.0674	0.1088	0.0151	0.062	0.084	0.0128
Dance	0.2705	0.1983	0.2504	0.2960	0.0980	0.143	0.229	0.1218
Face	0.0363	0.0314	0.0339	0.0436	0.0352	–	–	0.0179

Table 7:

e_{3d}

comparison on MoCap datasets [ 5 , 61 ] with trajectory-space and trajectory-prior NRSfM methods. Piecewise Bézier [ 2 ] and PPTA [ 4 ] are included as efficient trajectory-prior alternatives. Best performance for each sequence is highlighted in bold. PBezier and PPTA results are reported from their published papers because public code is unavailable; “–” denotes unavailable results.

Data	CSF1 [ 20 ]	CSF2 [ 22 ]	PTA [ 5 ]	NP [ 63 ]	Ours
Face Seq.1	0.5325	0.4677	0.1559	-	0.0774
Face Seq.2	0.9266	0.7909	0.1503	-	0.0442
Face Seq.3	0.5274	0.5474	0.1252	0.0394	0.0382
Face Seq.4	0.5392	0.5292	0.1348	0.0574	0.0522

Table 8:

e_{3d}

comparison on Dense Face Dataset with popular trajectory space [ 18 ] . Best performance for each sequence is highlighted in black.

Fixed-type camera motion
Data	CSF2 [ 22 ]	BMM [ 15 ]	PND [ 42 ]	CNS [ 43 ]	R-BMM [ 32 ]	TSM [ 57 ]	Ours
Eating2	0.2662	0.2084	0.2309	0.2250	0.1779	0.1767	0.1671
Smoking1	0.3534	0.2287	0.3397	0.2982	0.2290	0.2040	0.2181
Directions	0.2953	0.2872	0.3002	0.2674	0.2964	0.2688	0.2149
Smoking	0.1980	0.1759	0.4349	0.1968	0.1760	0.1693	0.1664
Waiting2	0.2125	0.1712	0.3306	0.1848	0.1478	0.1097	0.1161

Table 9:

e_{3d}

comparison on H3WB [ 70 ] with Fixed-type camera motion.

2^{\textrm{nd}}

best result is underlined.

Multi-circle-type camera motion
Data	CSF2 [ 22 ]	BMM [ 15 ]	PND [ 42 ]	CNS [ 43 ]	R-BMM [ 32 ]	TSM [ 57 ]	Ours
Eating2	0.0658	0.0384	0.0395	0.0605	0.0335	0.0276	0.0333
Smoking1	0.0507	0.0486	0.0358	0.0662	0.0332	0.0294	0.0325
Directions	0.0456	0.0569	0.0322	0.0535	0.0518	0.0315	0.0438
Smoking	0.0708	0.0308	0.0292	0.0346	0.0328	0.0282	0.0273
Waiting2	0.0892	0.0742	0.0450	0.0873	0.0637	0.0390	0.0424

Table 10:

e_{3d}

comparison on H3WB [ 70 ] with Multi-circle-type camera motion.

2^{\textrm{nd}}

best result is underlined.

Panel of Figure 7 — Figure 7: Constraint-satisfaction analysis on CMU MoCap sequences without ground-truth rotations. For each frame, we report the row-orthonormality residual $\|\mathbf{R}_{i}\mathbf{R}_{i}^{T}-\mathbf{I}_{2}\|_{F}$ of the recovered $2\times 3$ camera block after row-polar projection using Newton-Schulz inverse-square-root iterations. The plots show frame-wise residuals together with the sequence-level mean and maximum values. Across both the sequence, the residuals remain on the order of $10^{-7},$ confirming that the recovered camera blocks satisfy the orthographic Stiefel constraint to high numerical precision.


Methods	Subj. 07	Subj. 20	Subj. 23	Subj. 33	Subj. 34	Subj. 38	Subj. 39	Subj. 43	Subj. 93
C3DPO [ 49 ]	0.286	0.361	0.413	0.421	0.401	0.263	0.330	0.491	0.325
DNRSFM [ 29 ]	0.097	0.219	0.264	0.219	0.209	0.137	0.127	0.223	0.164
PR-PRN [ 66 ]	0.061	0.167	0.249	0.254	0.265	0.108	0.028	0.080	0.242
MHR-Net [ 67 ]	0.398	0.467	0.511	0.474	0.471	0.435	0.428	0.506	0.476
DNRSFM-R [ 17 ]	0.069	0.188	0.288	0.251	0.213	0.096	0.084	0.121	0.120

Table 11:

{e_{3d}}

on large CMU MOCAP (UNSEEN categories). Brown shows

2^{\textrm{nd}}

best.

Data	K-value	BMM [ 15 ]	R-BMM [ 32 ]	OPM [ 33 ]	TSM [ 57 ]
Drink	11	0.0164 0.0266	0.0084 0.0119	0.0083 0.0071	0.0041 0.0033
Pickup	12	0.0321 0.1731	0.0164 0.0198	0.0155 0.0152	0.0128 0.0126
Yoga	11	0.0284 0.1150	0.0122 0.0129	0.0140 0.0122	0.0114 0.0118
Stretch	13	0.0294 0.1034	0.0126 0.0144	0.0116 0.0124	0.0139 0.0131

Table 12: Suitability of our rotation solver when used in other high-performance methods. Bold indicate improved

e_{3d}

using our rotation estimates. For each method, Top row shows result using our rotation and Bottom row shows results reported by these methods using their rotation estimates.

Panel of Figure 8 — Figure 8: $e_{3d}$ variation w.r.t different noise levels in $\mathbf{W}$ .

Figure 9: Detailed computational pipeline of the proposed no-cone, no-dual approach. The rotation stage performs factor-side Euclidean correction using learnable per-sequence factors

\mathbf{Y}_{k}

and row-polar projection to recover valid camera blocks. The shape stage applies a fixed unrolled spectral-proximal loop consisting of reprojection-gradient updates, reshaping to

\mathbf{S}^{\sharp}

, weighted singular-value shrinkage, FISTA-style restart, and continuation. The main computations are matrix multiplications, SVD, shrinkage, and small row-polar projections.

Stage	Classical (SDP/ADMM)	Ours
Rotation estimation	$O(K^{6})$ per sequence (SDP)	$O(FK^{2})$
Shape estimation (per iter.)	$O(F(3P)^{2})+(3F)^{3}P$	$O(FP^{2})~\mathrm{if}~F>3P,~\mathrm{and}~O(F^{2}P)~\mathrm{if}~3P>F$
Linear solves / duals	required	none
Hardware acceleration	NA (CPU-bound)	excellent
Scales with frames $F$	superlinear	Regime-dependent: linear in $F$ , if $F>3P$ else quadratic
Scales with points $P$	quadratic	Regime-dependent: linear in $P$ , if $3P>F$ else quadratic

Table 13: Asymptotic complexity comparison between SOTA SDP and ADMM-based NRS f M and the proposed approach.

Dataset	Rotation time	Shape time	Total time
CMU MoCap Sequence
Drink	8.51	6.45	14.96
Pickup	7.79	4.88	12.67
Yoga	7.73	4.19	11.92
Stretch	7.83	4.21	12.04
Dance	8.57	6.39	14.96

Table 14: Stage-wise runtime of our solver on sparse MoCap and dense face benchmarks. Rotation time includes factor-side Euclidean correction and row-polar projection. Shape time includes the fixed-depth no-dual spectral-proximal solver after rotations are fixed. Total time is the combined solver runtime. All times are reported in seconds.

Sequence	$K=10$	$K=11$	$K=12$	$K=13$
Drink	0.0161	0.0166	0.0164	0.0153
Pickup	0.0272	0.0239	0.0267	0.0256
Yoga	0.0257	0.0148	0.0236	0.0207
Stretch	0.0122	0.0118	0.0119	0.0128

Table 15: Sensitivity of

e_{3d}

to the shape-space rank

K

on CMU MoCap sequences. The underlined entries indicate the

K

values used in the main-paper comparison following prior shape-space baselines, while bold entries indicate the best result in the sweep. The results show that the proposed solver is stable across nearby ranks and is not dependent on aggressive per-sequence rank tuning.

$\eta\downarrow$ /cap $\rightarrow$	0.90	0.95	0.99
1.02	(0.0660, 0.0211)	(0.0108, 0.0200)	(0.0112, 0.0177)
1.05	(0.1194, 0.0271)	(0.0640, 0.0265)	(0.0168, 0.0267)
1.10	(0.1369, 0.0577)	(0.1437, 0.0592)	(0.1248, 0.0591)

Table 16:

(e_{3d}^{\mathrm{Drink}},e_{3d}^{\mathrm{Pickup}})

with

N_{\textrm{pres}}=0

$\eta\downarrow$ /cap $\rightarrow$	0.90	0.95	0.99
1.02	(0.0510, 0.0233)	( 0.0108 , 0.0204)	(0.0111, 0.0176 )
1.05	(0.1219, 0.0280)	(0.0618, 0.0264)	( 0.0166 , 0.0267 )
1.10	(0.1360, 0.0698)	(0.1411, 0.0621)	(0.1234, 0.0590)

Table 17:

(e_{3d}^{\mathrm{Drink}},e_{3d}^{\mathrm{Pickup}})

with

N_{\textrm{pres}}=1

$\eta\downarrow$ /cap $\rightarrow$	0.90	0.95	0.99
1.02	(0.0976, 0.0238)	(0.0108, 0.0224)	(0.0113, 0.0186)
1.05	(0.1496, 0.0261)	(0.1205, 0.0274)	(0.0249, 0.0268)
1.10	(0.1690, 0.0880)	(0.1658, 0.0820)	(0.1570, 0.0826)

Table 18:

(e_{3d}^{\mathrm{Drink}},e_{3d}^{\mathrm{Pickup}})

with

N_{\textrm{pres}}=2

Sequence	Ours $\mathbf{R}$	GT $\mathbf{R}$	Gap
Drink	0.0166	0.0158	0.0008
Pickup	0.0267	0.0237	0.0030
Yoga	0.0148	0.0148	0.0000
Stretch	0.0127	0.0128	0.0001

Table 19: Pose-impact diagnostic on MoCap sequences with ground-truth rotations. We compare the 3d reconstruction results using estimated rotation and ground-truth rotation, respectively. The gap is computed as

\textrm{Gap}=|e_{3d}(\mathrm{Ours}~\mathbf{R})-e_{3d}(\mathrm{GT}~\mathbf{R})|

Figure 10: Qualitative results on real dense dataset sequence [ 18 ] .

Figure 11: Qualitative results on different categories present in H3WB dataset [ 57 ] .

Figure 12: Qualitative results on different categories in NRS f M challenge dataset [ 26 ] .

Figure 13: Qualitative results on CMU Mocap dataset [ 5 ] .

Figure 14: Qualitative results on Dense NRS f M Face dataset sequence [ 18 ] showing the $1^{\textrm{st}}$ order deformation field.

Deformable Shape Reconstruction

Deformable shape reconstruction estimates changing 3D geometry from images, video, touch, or embedded sensors.

See all 58 papers

Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis
Mar 20, 2026Jhacson Meza, Martin R. Oswald, Torsten SattlerCamera Pose Estimation Structure from Motion
Faculty of Electrical Engineering, Czech Technical University in Prague · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague · University of Amsterdam
Glob3R: Global Structure-from-Motion with 3D Foundation Models
Jul 10, 2026Junyuan Deng, Heng Li, Kejie Qiu +7Bundle Adjustment Multi-View 3D Reconstruction
The Hong Kong University of Science and Technology · Tongyi Lab, Alibaba Group · Nanjing University +1
RealDenseFace: Real-time Monocular 3D Face Reconstruction from Dense UV-space Priors
Aug 10, 2026Linzhou Li, Tianjia Shao, Kun ZhouStreaming 3D Reconstruction 3D Morphable Models
State Key Lab of CAD&CG, Zhejiang University

Mar 20, 2026J/K move · Enter open · S save

Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis

Jhacson Meza, Martin R. Oswald, Torsten Sattler

Faculty of Electrical Engineering, Czech Technical University in Prague · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague · University of Amsterdam

Camera Pose Estimation Structure from Motion Feed-Forward 3D Reconstruction Novel View Synthesis

Open paper AskPDF arXiv

Novel view synthesis (NVS) approaches such as NeRFs or 3DGS can produce photo-realistic 3D scene representation from a set of images with known extrinsic and intrinsic parameters. The necessary camera poses and calibrations are typically obtained from the images via Structure-from-Motion (SfM). Classical SfM approaches rely on local feature matches between the images to estimate both the poses and a sparse 3D model of the scene, using bundle adjustment to refine initial pose, intrinsics, and geometry estimates. In order to increase run-time efficiency, recent SfM systems forgo optimization via bundle adjustment. Instead, they train feed-forward (transformer-based) neural networks to directly regress camera parameters and the 3D structure. While orders of magnitude more efficient, such recent works produce significantly less accurate estimates. To stimulate research on developing SfM approaches that are both efficient and effective, this paper develops a benchmark focused on SfM for novel view synthesis. Using existing datasets and two simple strategies for making the reconstruction process more efficient, we show that: (1) simply using fewer features already significantly accelerates classical SfM methods while maintaining high pose accuracy. (2) using feed-forward networks to obtain initial estimates and refining them using classical SfM techniques leads to the best efficiency-effectiveness trade-off. Our benchmark and code are publicly available at https://github.com/jhacsonmeza/sfm-benchmark-nvs.

Abstract

Deformable Shape Reconstruction

Explore similar work

Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis

Glob3R: Global Structure-from-Motion with 3D Foundation Models

RealDenseFace: Real-time Monocular 3D Face Reconstruction from Dense UV-space Priors