GAUGE: Group-Wise View-Inconsistency Rectification for Feed-Forward 4D Tracking

Zhuoqian Feng; Weixing Chen; Ziliang Chen; Yang Liu; Liang Lin

cs.CVSep 26, 2026

GAUGE: Group-Wise View-Inconsistency Rectification for Feed-Forward 4D Tracking

Authors: Zhuoqian Feng, Weixing Chen, Ziliang Chen, Yang Liu, Liang Lin

Organizations: Sun Yat-sen University · Pengcheng Laboratory · X-Era AI Lab

Abstract

Feed-forward models regress dense 3D point trajectories directly from monocular video, yet the residual after global alignment is substantial and lacks a structural explanation. Measured on dynamic query points across models and datasets, the error concentrates along the view direction, while the scale correction each motion group requires differs. The predicted displacement direction nevertheless supports reliable grouping, with a median angle far below the 90° random baseline. The systematic part of the residual is therefore a family of radial degrees of freedom per motion group, along directions 2D observations cannot constrain. We call it group-wise view inconsistency. We present GAUGE (Group-wise Adaptive Unsupervised Gauge Estimation), a training-free and model-agnostic post-hoc module. It recovers motion groups from direction consistency and spatial connectivity, then estimates a per-frame radial scale and group-level translation from 1% to 5% metric anchors, four degrees of freedom per group and frame. On dynamic query points of eight trackers, including D4RT, 4RC and SM4RT, our correction lowers endpoint error by 15.1% to 62.6% over the uncorrected predictions, while spending the same anchors on gradient fine-tuning improves the same models by only -1.1% to 15.2%. Code is publicly available at https://github.com/HCPLab-SYSU/GAUGE.

Figures & tables

Figure 1: Per-group radial scale ambiguity. (a) RGB frame. (b) The prediction appears accurate in the camera view, but (c) seen from the side, it exposes a systematic depth bias along the view direction; (d) grouping recovers this bias as per-group radial scales, and (e) GAUGE removes it.

Figure 2: Error structure. Values in Tables A3 and A13 .

Parameterization	DOF ( $T$ frames)	EPE $\downarrow$
No correction	0	0.1812 / 0.1930
Single global scale	1	0.1835 / 0.1927
Per-group Sim(3)	7	0.1528 / 0.1983
GAUGE (ours)	$T+3$	0.1269 / 0.1597

Table 1: Parameterization comparison under the same grouping and anchors (D4RT).

Figure 3: Pipeline of GAUGE. Any tracker that outputs trajectories and visibility can plug in without retraining. The three blocks in GAUGE show main steps of correction.

Model	Method	PointOdyssey			EPE–Sim(3)
		EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$	Correction
	Base	0.1812 / 0.1930	0.6877 / 0.6654	0.1038 / 0.3358	—
D4RT	+GAUGE	0.1269 / 0.1597 ( +29.9 / +17.3 )	0.7648 / 0.7015 ( +11.2 / +5.4 )	0.1214 / 0.3320 ( +17.0 / -1.1)	+15.7 / -2.8
	Base	0.1421 / 0.2471	0.7334 / 0.6127	0.0711 / 0.2603	—
4RC	+GAUGE	0.1035 / 0.1470 ( +27.2 / +40.5 )	0.7832 / 0.7332 ( +6.8 / +19.7 )	0.0892 / 0.3172 ( +25.5 / +21.9 )	-29.6 / +12.8
	Base	0.1451 / 0.0778	0.7322 / 0.8792	0.1164 / 0.4478	—

Table 2: Main results on PointOdyssey. The Sim(3) column is the seven-degree-of-freedom fit under the same grouping and anchors. Cell values are DQS / Full; improvements (%) are in parentheses.

	EPE $\downarrow$			APD $\uparrow$			AJ $\uparrow$
Model	Base	+Ours	Impr. (%)	Base	+Ours	Impr. (%)	Base	+Ours	Impr. (%)
D4RT	0.1345	0.0838	+37.7	0.7783	0.8650	+11.1	0.0180	0.0207	+15.0
4RC	0.1531	0.1154	+24.6	0.7359	0.7970	+8.3	0.0128	0.0144	+12.5
V-DPM	0.1087	0.0758	+30.3	0.8082	0.8491	+5.1	0.0587	0.0646	+10.1
SM4RT	0.1256	0.0930	+26.0	0.7838	0.8331	+6.3	0.0154	0.0160	+3.9
TAPIP3D	0.0915	0.0623	+31.9	0.8351	0.8523	+2.1	0.0403	0.0452	+12.2

Table 3: WorldTrack results averaged over the four subsets under DQS. Averages are unweighted over the evaluated subsets; per-subset values are in Table A1 .

Figure 4: Qualitative comparison on two scenes. Query points are colored by signed depth error. Without grouping the errors scatter ahead of and behind the surface, while D4RT + GAUGE concentrates them better.

Grouping	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$
Oracle instance	0.1334 / 0.1159	0.7518 / 0.7717	0.1260 / 0.3913
Direction KNN only	0.1274 / 0.1606	0.7692 / 0.7006	0.1210 / 0.3328
Spatial merge only	0.1271 / 0.1602	0.7693 / 0.6992	0.1213 / 0.3322
Full (Ours)	0.1269 / 0.1597	0.7648 / 0.7015	0.1214 / 0.3320

Table 4: Ablation studies (D4RT) under the camera-center parameterization; the 4RC version is in Appendix Table A10 .

Anchor source	Median error (cm)	Radial fraction	$\Delta$ EPE
Ground truth	0	—	29.9%
Visual SLAM keyframes	8.57	0.49	20.6%
Single-frame depth	8.50	1	10.1%
Sparse LiDAR returns	0.81	1	29.2%

Table 5: Effect of anchor source on the gain (D4RT).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

DQS protocol
Model	Method	ADT			PO			PStudio			DS
		EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$
D4RT	Base	0.1516	0.7462	0.0398	0.1770	0.7191	0.0187	0.1092	0.8141	0.0094	0.1000	0.8338	0.0039
	+Ours	0.1142	0.8025	0.0450	0.1001	0.8152	0.0216	0.0526	0.9482	0.0120	0.0681	0.8941	0.0043
	Impr. (%)	+24.7	+7.5	+13.1	+43.4	+13.4	+15.5	+51.8	+16.5	+27.7	+31.9	+7.2	+10.3
4RC	Base	0.2350	0.6687	0.0218	0.1587	0.6944	0.0161	0.1269	0.7955	0.0089	0.0917	0.7850	0.0043

Appendix

Table A1: Per-subset WorldTrack results under the DQS and Full protocols, in the world coordinate frame. ADT and PStudio provide no depth ground truth, so the three depth-dependent models are marked with a dash. Improvements are computed from the ratio of the absolute values in this table. Table 3 in the main text reports the averages over the subsets under DQS.

Model	Status	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$	$\Delta$ EPE (%)
D4RT	base	0.1812 / 0.1930	0.6877 / 0.6654	0.1038 / 0.3358	—
	+TTO	0.1720 / 0.1685	0.7090 / 0.6933	0.1057 / 0.3380	5.1 / 12.7
	+Ours	0.1269 / 0.1597	0.7648 / 0.7015	0.1214 / 0.3320	29.9 / 17.3
4RC	base	0.1421 / 0.2471	0.7334 / 0.6127	0.0711 / 0.2603	—
	+TTO	0.1419 / 0.2436	0.7358 / 0.6153	0.0863 / 0.2611	0.1 / 1.4
	+Ours	0.1035 / 0.1470	0.7832 / 0.7332	0.0892 / 0.3172	27.2 / 40.5

Appendix

Table A2: Comparison with test-time optimization. Three metrics and the relative change in endpoint error for five models under three settings, base, +TTO and +Ours.

Model	Radial energy (%)	Flow direction error ( ∘ )	Between-group scale variance (%)
D4RT	77.7 / 74.8	24.0 / 19.8	62.9 / 74.4
4RC	67.3 / 89.4	38.4 / 31.4	73.7 / 72.2
V-DPM	73.2 / 73.0	35.5 / 37.9	47.6 / 69.7
SM4RT	62.3 / 80.2	64.1 / 62.9	47.9 / 68.1
TAPIP3D	67.9 / 37.9	7.4 / 87.7	59.7 / 15.1
SpatialTrackerV2	37.6 / 44.7	87.8 / 67.0	26.7 / 30.4

Appendix

Table A3: Error structure of the eight trackers. The isotropic baseline of radial energy is 50% and the random baseline of flow direction error is about

90^{\circ}

Noise $\sigma$ (cm)	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$	$\Delta$ EPE vs. clean (%)
base	0.1812 / 0.1930	0.6877 / 0.6654	0.1038 / 0.3358	—
0 (clean)	0.1269 / 0.1597	0.7648 / 0.7015	0.1214 / 0.3320	0.0000 / 0.0000
1	0.1277 / 0.1598	0.7645 / 0.7040	0.1216 / 0.3313	-0.6524 / -0.1072
5	0.1386 / 0.1722	0.7502 / 0.6944	0.1210 / 0.3234	-9.2242 / -7.8694
10	0.1549 / 0.1994	0.7248 / 0.6634	0.1198 / 0.3122	-22.0386 / -24.9011
20	0.2241 / 0.2634	0.6565 / 0.6033	0.1107 / 0.2872	-76.5662 / -64.9686

Appendix

Table A4: Robustness to anchor noise (D4RT). The correction problem is re-solved after applying isotropic Gaussian noise of standard deviation

\sigma

to the ground-truth anchor positions; the base row is the uncorrected baseline and

\sigma=0

is the noiseless anchor.

Model	Protocol	1%	2%	5%	10%	20%	50%	100%
D4RT	DQS	0.1399	0.1371	0.1269	0.1312	0.1216	0.1167	0.1289
D4RT	Full	0.1704	0.1702	0.1597	0.1495	0.1326	0.1300	0.1363
4RC	DQS	0.1048	0.1124	0.1035	0.1004	0.1177	0.1052	0.1095
4RC	Full	0.1591	0.1510	0.1470	0.1347	0.1467	0.1253	0.1257
V-DPM	DQS	0.1217	0.0981	0.1086	0.1000	0.1014	0.1115	0.1150
V-DPM	Full	0.0669	0.0641	0.0619	0.0626	0.0595	0.0611	0.0566

Appendix

Table A5: Anchor budget. Endpoint error at each budget level; Appendix Figure A3 plots this curve for D4RT.

Model	Grouping	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$
D4RT	Oracle instance	0.1334 / 0.1159	0.7518 / 0.7717	0.1260 / 0.3913
D4RT	Unsupervised (Ours)	0.1269 / 0.1597	0.7648 / 0.7015	0.1214 / 0.3320
4RC	Oracle instance	0.1091 / 0.1216	0.7655 / 0.7435	0.0777 / 0.3213
4RC	Unsupervised (Ours)	0.1035 / 0.1470	0.7832 / 0.7332	0.0892 / 0.3172
V-DPM	Oracle instance	0.1194 / 0.0549	0.7871 / 0.9071	0.1043 / 0.4674
V-DPM	Unsupervised (Ours)	0.1086 / 0.0619	0.8091 / 0.9077	0.1164 / 0.4634

Appendix

Table A6: Unsupervised grouping versus oracle instance grouping. On PointOdyssey, the three metrics of the eight trackers when the correction switches from oracle instances to the unsupervised motion grouping of this paper; the Unsupervised row is the result of applying GAUGE in Table 2 of the main text. Instance labels are used only in the Oracle instance row.

Model	$\Delta$ EPE (%)	Resampling std (%)	Paired bootstrap 95% CI (%)
D4RT	28.70	1.0	[18.2, 39.3]
4RC	19.7	3.8	[16.8, 40.2]
V-DPM	18.6	2.4	[8.0, 28.8]
SM4RT	20.4	8.3	[5.4, 36.2]
TAPIP3D	31.2	1.2	[17.4, 45.8]
SpatialTrackerV2	9.4	3.2	[-3.1, 20.8]

Appendix

Table A7: Resampling dispersion. These values use a different estimator from the improvement in Table 2 of the main text and are not directly comparable.

Model	P25 (%)	P50 (%)	P75 (%)	Seq. with gain (%)
D4RT	19.3	31.0	41.7	92.3
4RC	-16.9	36.6	50.8	66.7
V-DPM	-3.6	18.4	37.5	53.8
SM4RT	-3.9	22.7	38.4	66.7
DELTA	43.8	60.2	65.2	100.0
TAPIP3D	8.7	28.6	65.0	91.7

Appendix

Table A8: Distribution of per-sequence endpoint error improvement. The quantiles use a different estimator from the improvement in Table 2 of the main text and are not directly comparable.

Median wall (s)	P95 wall (s)	Median CPU (s)	Peak RSS (MiB)	Extra RSS (MiB)
0.490 / 2.601	3.181 / 35.183	0.490 / 2.600	594.109 / 769.656	0.742 / 29.688

Appendix

Table A9: Post-processing cost. Timings in seconds per sequence, from a single core whose frequency scales dynamically between 1.50 and 3.71 GHz; each cell gives DQS / Full. No GPU, autograd or optimizer is used.

Grouping Method	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$
No grouping (global)	0.1370 / 0.2427	0.7511 / 0.6217	0.0716 / 0.2651
Oracle instance	0.1091 / 0.1216	0.7655 / 0.7435	0.0777 / 0.3213
Full (Ours)	0.1035 / 0.1470	0.7832 / 0.7332	0.0892 / 0.3172

Appendix

Table A10: Ablations repeated on 4RC. Configurations are taken from Table 4 in the main text. The two forms that scale about an in-group anchor were not repeated on 4RC, and the single-form-on-all-groups and pure-translation rows report endpoint error only. In (c), AE is the average per-anchor efficiency accumulated to that budget, defined as in Table A5 .

Configuration	EPE $\downarrow$	APD $\uparrow$	AJ $\uparrow$
TAPIP3D (GT depth input)	0.1548	0.7600	0.1560
TAPIP3D (GT depth input) + Ours	0.0966	0.8365	0.1576
TAPIP3D (predicted depth input)	1.6705	0.0565	0.0319
TAPIP3D (predicted depth) + Ours	0.6949	0.3639	0.0651

Appendix

Table A11: Depth input versus post-hoc correction. Three metrics of TAPIP3D before and after correction under two input conditions, ground-truth depth and predicted depth; DQS only.

Model	Predicted-motion gain %	GT-motion gain %	$\Delta$ (pp)	Jaccard
D4RT	30.5	30.5	-0.01	0.7
4RC	16.5	37.6	-21.11	0.59
V-DPM	23.6	29.4	-5.88	0.71
SM4RT	83.4	24.6	58.82	0.16
SpatialTrackerV2	30.4	36.1	-5.69	0.35
TAPIP3D	34.5	38.3	-3.80	0.61

Appendix

Table A12: Screening control with ground-truth motion magnitude. Gain in endpoint error after redoing the dynamic query screening with ground-truth motion magnitude in place of the predicted value;

\Delta

is the predicted-motion gain minus the ground-truth one and Jaccard is the overlap of the two selected point sets. These gains are computed on motion-screened point sets and are not comparable with the DQS improvement in Table 2 of the main text.

Model	Protocol	Observed	Null mean	Null p95	Excess over p95
D4RT	DQS	62.94	12.39	30.97	28.24
D4RT	Full	74.36	11.03	12.38	51.09
4RC	DQS	73.65	12.81	24.09	48.54
4RC	Full	72.17	4.52	12.8	60.92
V-DPM	DQS	47.64	16.83	32.52	24.05
V-DPM	Full	69.67	4.42	7.17	47.95

Appendix

Table A13: Null hypothesis test. The tested statistic is the between-group scale variance, that is, the between-group difference in per-point optimal radial scale that unsupervised grouping can explain; the definition is given in Appendix A.3 . The null model keeps the size of each group unchanged and randomly permutes the point-to-group labels, one hundred times per sequence.

Figure A1: Failure prediction. Relative endpoint error improvement against between-group scale variance, one point per model, split and protocol. The gray line is the least-squares fit over the 58 points and the band its 95% confidence interval.

Figure A2: Heatmap of relative endpoint error improvement.

Model	Protocol	Baseline	GAUGE 5%	Ceiling
D4RT	DQS	0.1812	0.1269	0.1167
D4RT	Full	0.1930	0.1597	0.1300
4RC	DQS	0.1421	0.1035	0.1004
4RC	Full	0.2471	0.1470	0.1203
V-DPM	DQS	0.1451	0.1086	0.0994
V-DPM	Full	0.0778	0.0619	0.0566

Appendix

Table A14: Error budget. The upper bound from fitting radial scale and translation per group and per frame on all ground-truth points, against what GAUGE achieves with 5% anchors.

Figure A3: Anchor budget of D4RT. (a) Endpoint error and (b) average accumulated per-anchor efficiency.

3D Point Tracking

3D point tracking estimates how scene points move through space and time, even under camera motion, occlusion, and deformation.

See all 30 papers

Point4D: Long-range 4D Motion Reconstruction
Sep 8, 2026Minsik Jeon, Jay Karhade, Deva Ramanan +1Point Tracking 3D Point Tracking
Carnegie Mellon University
Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction
May 29, 2026Mungyeom Kim, Minkyeong Jeon, Honggyu An +10Point Tracking 3D Gaussian Splatting
KAIST AI · ETH Zürich, ETH AI Center · Sony AI +1
No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos
May 21, 2026Matteo Balice, Yanik Kunzi, Chenyangguang Zhang +33D Gaussian Splatting 4D Reconstruction
Politecnico di Milano · ETH Zürich · ETH AI Center

Sep 8, 2026J/K move · Enter open · S save

Point4D: Long-range 4D Motion Reconstruction

Minsik Jeon, Jay Karhade, Deva Ramanan, Shubham Tulsiani

Carnegie Mellon University

Point Tracking 3D Point Tracking 4D Reconstruction Dynamic Scene Reconstruction Multi-Object Tracking

Open paper AskPDF arXiv

We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io

Abstract

3D Point Tracking

Explore similar work

Point4D: Long-range 4D Motion Reconstruction

Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction

No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos