What Makes a Latent Space Shared? Direct and Recoverable Compatibility in Motion VAEs

Iago Brito; Walcy Rios; Lucas Fernandes; Kaiki de Oliveira; Arlindo Galvão Filho

cs.AIOct 12, 2026

What Makes a Latent Space Shared? Direct and Recoverable Compatibility in Motion VAEs

Authors: Iago Brito, Walcy Rios, Lucas Fernandes, Kaiki de Oliveira, Arlindo Galvão Filho

Organizations: Advanced Knowledge Center in Immersive Technologies (AKCIT) Federal University of Goiás (UFG)

Abstract

A shared latent space is often assumed to make representations from different schemas directly interchangeable, yet sharing parameters, dimensionality, or a prior does not determine how corresponding instances are expressed in latent coordinates. We show that this assumption conflates recoverable correspondence, direct compatibility, and decoder reuse. In a controlled multi-schema motion VAE on HumanML3D, even identical inputs yield only $5.9$ -- $17.7\%$ raw Recall@1 and $0.221$ -- $0.484$ paired-token cosine, despite correspondence being almost perfectly recoverable after linear calibration. Cycle consistency strengthens recoverability but still leaves direct retrieval and token agreement near zero. In contrast, only $64$ paired clips raise raw Recall@1 to $88.4$ -- $90.3\%$ and token cosine to approximately $0.82$ , making most post-hoc alignment unnecessary, although crossed decoding remains substantially worse than native reconstruction. We further find that reducing measured marginal discrepancy does not imply schema invariance. These results show that a shared backbone or prior does not ensure directly compatible latent coordinates, while limited paired correspondence can substantially improve direct compatibility in the tested setting.

Figures & tables

Figure 1: Multi-schema variational autoencoder. Each motion view retains its feature representation and connects to a shared backbone through schema-specific input and output adapters. All schemas share the variational autoencoder and use the same latent dimensionality.

Condition	VAE objective	Additional signal
Floor	$\mathcal{L}_{\mathrm{floor}}$	None
Cycle	$\mathcal{L}_{\mathrm{floor}}+\lambda_{z}\mathcal{L}_{z\mathrm{cyc}}+\lambda_{x}\mathcal{L}_{x\mathrm{cyc}}$	Round trips
MMD	$\mathcal{L}_{\mathrm{floor}}+\lambda_{m}\mathcal{L}_{\mathrm{MMD}}$	Latent marginals
Discriminator	$\mathcal{L}_{\mathrm{floor}}-\lambda_{d}\mathcal{L}_{\mathrm{dom}}$	Schema identity
Anchor	$\mathcal{L}_{\mathrm{floor}}+\lambda_{a}\mathcal{L}_{\mathrm{anchor}}$	Motion pairs

Table 1: Training conditions and their additional cross-schema signals. The discriminator row gives the VAE-side objective, while the domain classifier minimizes

\mathcal{L}_{\mathrm{dom}}

View pair	Schema A	Schema B	Controlled difference
Identical	native, 263-d	identical, 263-d	none
Reordered	native, 263-d	permuted, 263-d	joint/block order
Positions	native, 263-d	positions, 71-d	feature selection
Rotations	native, 263-d	rotations, 134-d	feature selection
Sparse	native, 263-d	sparse, 41-d	joint set and frame subsampling
Disjoint	positions, 63-d	rotations, 126-d	disjoint feature blocks

Table 2: Controlled views of the same HumanML3D motions. Schema A is native HumanML3D except in Disjoint.

View pair	Raw	+ Procrustes	+ Ridge	+ CCA-16
Identical	5.91–17.72	96.30–98.36	97.88–99.11	99.64–99.86
Reordered	0.30–36.82	94.37–98.61	97.54–99.04	99.77–99.91
Positions	0.11–0.94	58.65–61.95	60.74–63.59	95.78–96.10
Rotations	0.16–1.64	78.85–82.66	82.89–87.66	95.78–96.30
Sparse	0.21–0.36	44.07–47.35	45.28–47.38	90.90–91.95
Disjoint	0.14–0.30	19.32–21.26	28.60–31.32	55.66–57.62

Table 3: Floor retrieval across three training seeds per view pair. Each experiment has two interfaces sharing one backbone. Maps are fitted separately to all

20{,}942

eligible training pairs. Entries are Recall@1 percentages; Disjoint uses positional schema A.

Condition	Raw	+ Procrustes	+ Ridge	+ CCA-16
Floor	0.16–1.64	78.85–82.66	82.89–87.66	95.78–96.30
Cycle	0.09–0.16	94.25–94.89	95.80–96.37	99.16–99.25
MMD	0.23–2.85	80.18–83.07	84.24–87.18	97.08–97.67
Discriminator	12.55–44.14	82.66–85.26	85.77–88.59	96.58–97.42

Table 4: Unpaired objectives on Rotations across three training seeds. Entries are Recall@1 percentages, with each calibration fitted separately to all eligible training pairs.

Figure 2: Paired supervision and decoder reuse. (a) Rotations Recall@1 versus paired clip identities. (b) Native and crossed local MPJPE on the same

512

clips at matched checkpoints. Lines and markers summarize three runs as described in the text.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Comparison	Schema A	Schema B	B width
Identical	native	Native features in original order	263
Reordered	native	Permuted joints and feature blocks	263
Positions	native	ROOT, POS, CONTACT	71
Rotations	native	ROOT, ROT, CONTACT	134
Sparse	native	ROOT, 11 non-root positions, CONTACT; every second frame	41
Disjoint	POS, 63-d	ROT without root or contact features	126

Appendix

Table 5: Controlled schema definitions. Each row defines a separate two-interface experiment.

Module	Layers
Input adapter	Linear $d_{i}\!\rightarrow\!256$ , ReLU, Linear $256\!\rightarrow\!256$
Shared encoder	Two width-256 residual blocks, followed by parallel mean and log-variance heads $256\!\rightarrow\!64$
Shared decoder	Linear $64\!\rightarrow\!256$ , followed by two width-256 residual blocks
Output adapter	Linear $256\!\rightarrow\!256$ , ReLU, Linear $256\!\rightarrow\!d_{i}$
Domain classifier	Linear $128\!\rightarrow\!128$ , LeakyReLU, Linear $128\!\rightarrow\!128$ , LeakyReLU, Linear $128\!\rightarrow\!2$

Appendix

Table 6: Main MLP architecture, with hidden width

256

and latent dimension

64

View pair	Seed	Raw	Proc.	Ridge	CCA
Identical	42	10.65	98.36	99.11	99.86
Identical	1	5.91	96.30	97.88	99.64
Identical	2	17.72	98.02	98.77	99.86
Reordered	42	0.30	94.37	97.54	99.79
Reordered	1	1.46	98.15	98.77	99.77
Reordered	2	36.82	98.61	99.04	99.91

Appendix

Table 7: Floor Recall@1 (%) across controlled views and training seeds. Retrieval is full-gallery

B\rightarrow A

; Disjoint uses positions as schema A.

Condition	Seed	Raw	Proc.	Ridge	CCA
Cycle	42	0.11	94.37	95.80	99.16
Cycle	1	0.16	94.89	96.37	99.20
Cycle	2	0.09	94.25	95.94	99.25
MMD	42	0.23	83.07	87.18	97.08
MMD	1	0.87	82.07	86.09	97.22
MMD	2	2.85	80.18	84.24	97.67

Appendix

Table 8: Rotations objective comparisons across training seeds. Entries are Recall@1 (%). MMD uses

\lambda_{m}=1

; full-pair Anchor achieves

100\%

across all three seeds.

Condition	Raw	Proc.	Ridge	CCA
Floor	0.14–0.30	19.32–21.26	28.60–31.32	55.66–57.62
Cycle	0.14–0.32	50.05–55.02	60.47–63.05	77.08–78.49
Anchor, all pairs	99.59–99.73	99.61–99.75	99.70–99.82	99.16–99.57

Appendix

Table 9: Disjoint Recall@1 (%) across three training seeds. Ranges are min–max; calibration uses all eligible training pairs.

View	Map	Pairs	Trained Floor	Random encoder	$\Delta$ R@1
Rotations	Proc.	256	78.49 (75.57–78.70)	24.04 (22.31–26.69)	+54.45
Rotations	Proc.	20,942	82.30 (78.85–82.66)	36.68 (33.10–38.18)	+45.62
Rotations	CCA	256	67.24 (67.04–69.53)	6.84 (6.64–7.82)	+60.40
Rotations	CCA	20,942	96.24 (95.78–96.30)	73.63 (72.15–73.79)	+22.60
Disjoint	Proc.	256	14.80 (13.96–15.31)	4.43 (2.01–5.09)	+10.38
Disjoint	Proc.	20,942	19.73 (19.32–21.26)	9.97 (6.41–10.45)	+9.76

Appendix

Table 10: Training-only calibration with limited and full correspondence budgets. Values are median Recall@1 (%) with min–max ranges across three seeds.

\Delta

R@1 denotes the difference between trained and random medians.

Condition	Raw R@1 $\uparrow$	Cosine $\uparrow$	Rel. $L_{2}$ $\downarrow$
Identical Floor	5.91–17.72	0.221–0.484	0.989–1.227
Reordered Floor	0.30–36.82	0.084–0.706	0.706–1.338
Rotations Floor	0.16–1.64	0.114–0.233	1.224–1.319
Rotations Cycle	0.09–0.16	$-0.025$ –0.026	1.391–1.427
Rotations Anchor-64	88.41–90.33	0.824–0.826	0.562–0.566

Appendix

Table 11: Direct token agreement and raw clip Recall@1 (%) across three seeds. The mismatched-identity reference ranges from

-0.057

0.019

in cosine similarity and

1.388

1.447

in relative

L_{2}

Condition	Raw R@1 $\uparrow$	Probe acc. $\downarrow$	MMD 2 $\downarrow$
Floor	0.16–1.64	99.91–99.97	13.04–15.78
Cycle	0.09–0.16	99.93–99.98	50.76–63.88
MMD ( $\lambda_{m}=1$ )	0.23–2.85	99.95–99.99	6.14–9.81
Discriminator	12.55–44.14	99.94–100.00	16.40–21.54
Anchor (all pairs)	100.00	99.93–100.00	0.36–0.48

Appendix

Table 12: Marginal diagnostics on Rotations across three seeds. Lower probe accuracy indicates weaker schema separability; lower MMD 2 indicates smaller measured distributional discrepancy. MMD 2 is reported in units of

10^{-3}

Clip pairs	Selected step	Selected R@1	Final R@1
16	2,000	18.23–25.82	62.00–75.75
64	4,000	85.13–86.20	88.41–90.33
256	6,000	98.91–99.20	94.07–94.98
1,024	12,000–20,000	99.89–100.00	99.02–99.32

Appendix

Table 13: Checkpoint sensitivity for nested Rotations Anchor budgets. Recall@1 (%) is reported as min–max across three seeds. Primary limited-pair results use the final checkpoint at

200{,}000

steps.

Identities	Fit unit	Proc.	Ridge	CCA	Anchor raw
16	Clip	10.36–15.56	15.40–16.97	4.56–5.82	62.00–75.75
16	Window	38.32–43.57	26.48–31.20	0.02–0.07	62.00–75.75
64	Clip	62.16–65.99	29.20–43.34	41.51–48.59	88.41–90.33
64	Window	66.99–70.46	62.02–70.96	65.42–70.05	88.41–90.33
256	Clip	75.48–79.20	77.69–83.90	62.18–71.88	94.07–94.98
256	Window	76.25–80.54	79.06–84.35	88.30–89.90	94.07–94.98

Appendix

Table 14: Rotations Recall@1 (%) under matched paired-identity budgets. Calibration uses three Floor seeds; Anchor uses three independent runs at fixed final checkpoints. Ranges are min–max.

Condition	Raw	Proc.	CCA
Floor	0.32	82.66	96.30
Latent cycle only	0.14	88.62	98.08
Motion cycle only	0.09	83.90	97.49
Both cycles	0.11	94.37	99.16

Appendix

Table 15: Cycle-component ablation on Rotations (seed 42). Full-gallery Recall@1 (%); maps use all training correspondences.

Condition	Backbone	Raw	Proc.	CCA
Floor	MLP	0.16–1.64	78.85–82.66	95.78–96.30
Floor	TCN	0.16–1.98	93.77–94.48	99.50–99.82
Cycle	MLP	0.09–0.16	94.25–94.89	99.16–99.25
Cycle	TCN	0.07–0.30	97.31–98.04	99.82–99.91
Anchor, all	MLP	100.00	100.00	100.00
Anchor, all	TCN	100.00	100.00	99.98–100.00

Appendix

Table 16: Backbone robustness on Rotations. Full-gallery Recall@1 (%), reported as min–max over three seeds. The architectures are not parameter-matched.

Condition	Raw R@1	Native MPJPE	Crossed MPJPE
Identical Floor	5.91–17.72	42.99–43.36	115.20–199.51
Reordered Floor	0.30–36.82	42.93–43.56	65.47–214.70
Rotations Floor	0.16–1.64	42.98–43.72	198.50–221.52
Rotations Cycle	0.09–0.16	33.08–33.35	433.50–509.66
Rotations Anchor-64	88.41–90.33	43.38–44.45	73.87–79.83

Appendix

Table 17: Direct decoder compatibility. Raw Recall@1 (%) and local MPJPE (mm), reported as min–max across three seeds.

Condition	Native	Direct cross	Ridge cross
Floor	43.72	205.59	92.08
Cycle	33.16	496.71	82.28
Anchor-64	43.38	74.23	72.48

Appendix

Table 18: Token-calibrated decoding on Rotations (seed 42). Local MPJPE (mm). The ridge map uses

200{,}000

paired training frames from

6{,}250

clips.

Latent Space Alignment

Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.

See all 76 papers

Mar 24, 2026

MoCHA: Denoising Caption Supervision for Motion-Text Retrieval

Text-motion retrieval systems learn shared embedding spaces from motion-caption pairs via contrastive objectives. However, each caption is not a deterministic label but a sample from a distribution of valid descriptions: different annotators produce different text for the same motion, mixing motion-recoverable semantics (action type, body parts, directionality) with annotator-specific style and inferred context that cannot be determined from 3D joint coordinates alone. Standard contrastive training treats each caption as the single positive target, overlooking this distributional structure and inducing within-motion embedding variance that weakens alignment. We propose MoCHA, a text canonicalization framework that reduces this variance by projecting each caption onto its motion-recoverable content prior to encoding, producing tighter positive clusters and better-separated embeddings. Canonicalization is a general principle: even deterministic rule-based methods improve cross-dataset transfer, though learned canonicalizers provide substantially larger gains. We present two learned variants: an LLM-based approach (GPT-5.2) and a distilled FlanT5 model requiring no LLM at inference time. MoCHA operates as a preprocessing step compatible with any retrieval architecture. Applied to MoPa (MotionPatches), MoCHA sets a new state of the art on both HumanML3D (H) and KIT-ML (K): the LLM variant achieves 13.9% T2M R@1 on H (+3.1pp) and 24.3% on K (+10.3pp), while the LLM-free T5 variant achieves gains of +2.5pp and +8.1pp. Canonicalization reduces within-motion text-embedding variance by 11-19% and improves cross-dataset transfer substantially, with H to K improving by 94% and K to H by 52%, demonstrating that standardizing the language space yields more transferable motion-language representations.

Nikolai Warner, Cameron Ethan Taylor, Irfan Essa +1

Georgia Institute of Technology, Atlanta, GA, USA · Amazon, Sunnyvale, CA, USA

Cross-Modal Retrieval Multimodal Contrastive Learning

Ask PDF

Mar 23, 2026

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture. Existing unified models handle only restricted modality subsets (e.g., Motion-Text or static Pose-Image) and predominantly rely on discrete tokenization, which introduces quantization errors and disrupts temporal continuity. UniMotion overcomes both limitations through a core principle: treating motion as a first-class continuous modality on equal footing with RGB. A novel Cross-Modal Aligned Motion VAE (CMA-VAE) and symmetric dual-path embedders construct parallel continuous pathways for Motion and RGB within a shared LLM backbone. To inject visual-semantic priors into motion representations without requiring images at inference, we propose Dual-Posterior KL Alignment (DPA), which distills a vision-fused encoder's richer posterior into the motion-only encoder. To address the cold-start problem -- where text supervision alone is too sparse to calibrate the newly introduced motion pathway -- we further propose Latent Reconstruction Alignment (LRA), a self-supervised pre-training strategy that uses dense motion latents as unambiguous conditions to co-calibrate the embedder, backbone, and flow head, establishing a stable motion-aware foundation for all downstream tasks. UniMotion achieves state-of-the-art performance across seven tasks spanning any-to-any understanding, generation, and editing among the three modalities, with especially strong advantages on cross-modal compositional tasks.

Ziyi Wang, Xinshun Wang, Shuang Chen +2

State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology

Cross-Modal Learning Unified Multimodal Models Cross-Modal Alignment Human Motion Generation Human Motion Understanding Multimodal Foundation Models+1

Ask PDF

Oct 14, 2026

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.

Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +3

Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1

3D Scene Generation 3D Scene Representation 3D-Aware Video Generation 3D Reconstruction

Ask PDF

Abstract

Latent Space Alignment

Explore similar work

MoCHA: Denoising Caption Supervision for Motion-Text Retrieval

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space