MoCHA: Denoising Caption Supervision for Motion-Text Retrieval

Nikolai Warner; Cameron Ethan Taylor; Irfan Essa; Apaar Sadhwani

cs.CVMar 24, 2026

MoCHA: Denoising Caption Supervision for Motion-Text Retrieval

Authors: Nikolai Warner, Cameron Ethan Taylor, Irfan Essa, Apaar Sadhwani

Organizations: Georgia Institute of Technology, Atlanta, GA, USA · Amazon, Sunnyvale, CA, USA

Abstract

Text-motion retrieval systems learn shared embedding spaces from motion-caption pairs via contrastive objectives. However, each caption is not a deterministic label but a sample from a distribution of valid descriptions: different annotators produce different text for the same motion, mixing motion-recoverable semantics (action type, body parts, directionality) with annotator-specific style and inferred context that cannot be determined from 3D joint coordinates alone. Standard contrastive training treats each caption as the single positive target, overlooking this distributional structure and inducing within-motion embedding variance that weakens alignment. We propose MoCHA, a text canonicalization framework that reduces this variance by projecting each caption onto its motion-recoverable content prior to encoding, producing tighter positive clusters and better-separated embeddings. Canonicalization is a general principle: even deterministic rule-based methods improve cross-dataset transfer, though learned canonicalizers provide substantially larger gains. We present two learned variants: an LLM-based approach (GPT-5.2) and a distilled FlanT5 model requiring no LLM at inference time. MoCHA operates as a preprocessing step compatible with any retrieval architecture. Applied to MoPa (MotionPatches), MoCHA sets a new state of the art on both HumanML3D (H) and KIT-ML (K): the LLM variant achieves 13.9% T2M R@1 on H (+3.1pp) and 24.3% on K (+10.3pp), while the LLM-free T5 variant achieves gains of +2.5pp and +8.1pp. Canonicalization reduces within-motion text-embedding variance by 11-19% and improves cross-dataset transfer substantially, with H to K improving by 94% and K to H by 52%, demonstrating that standardizing the language space yields more transferable motion-language representations.

Figures & tables

Panel of Figure 1 — Figure 1: Each caption is a different sample from a distribution of valid descriptions. Three annotators describe the same motion (top) with different captions, each mixing motion-recoverable semantics $s$ ( blue ) with annotator-specific nuisance factors $a$ ( red )—stylistic variation. Standard contrastive training treats each as the single correct target; MoCHA projects each onto $s$ , producing a single deterministic positive.

Panel of Figure 2 — Figure 2: MoCHA overview. (a) Motivated by the $(s,a)$ decomposition (Section 3.1 ), $C(\cdot)$ projects each caption onto $s$ by stripping stylistic variation $a$ ( red ). $C$ is implemented via LLM and distilled into FlanT5 for LLM-free inference. (b) Blend training balances both views: the denoised $C(t_{i})$ anchors embeddings around $s$ to reduce gradient variance, while the original $t_{i}$ regularizes for natural-language queries.

Dataset	$N$	Original	MoCHA (T5)	$\Delta$
HumanML3D	4,344	0.587	0.522	$-$ 11.1%
KIT-ML	210	0.561	0.456	$-$ 18.7%

Table 1: Within-motion text embedding variance (C4). Canonicalization reduces the spread of caption embeddings for the same motion by 11–19%, confirming that

a

introduces measurable noise into the contrastive positive.

V(m)

: mean pairwise cosine dissimilarity across a motion’s

K

captions in the baseline encoder’s 256-dim space.

HumanML3D
	Model	Intra	Align	Inter NN	Sep
	Baseline	0.413	0.601	0.976	0.423
	MoCHA	0.444	0.636	0.971	0.457
KIT-ML
	Baseline	0.456	0.515	0.941	0.484
	MoCHA	0.566	0.553	0.938	0.604

Table 2: Embedding space geometry (C4): baseline vs. MoCHA in the trained encoder’s 256-dim space. By removing

a

from the training signal, MoCHA produces embeddings where same-motion captions cluster more tightly (Intra) and align more closely with their motion (Align), improving separation by +8–25%. This confirms that input-level variance reduction propagates to a better-structured retrieval space. Sep: Intra / Inter NN. Full breakdown in Appendix 0.J .

Panel of Figure 3 — Figure 3: Canonicalization projects captions onto $s$ , improving retrieval. Top row (colored): ground truth; bottom row (gray): baseline rank-1 error. (a) Verbose $a$ buries the action; MoCHA extracts $s$ while preserving the metaphor. (b) Annotator uncertainty ( $a$ ); canonicalization extracts shared kinematic content. (c) Complex description decomposed into sequential $s$ , disambiguating from similar motions. (d) Over-specified caption dilutes the contrastive signal; MoCHA strips $a$ , retains discriminative $s$ .

HumanML3D
	Text to motion				Motion to text
Methods	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	MedR $\downarrow$	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	MedR $\downarrow$
TEMOS	2.12	8.26	13.52	173.0	3.86	9.38	14.00	183.25
T2M	1.80	7.12	12.47	81.00	2.92	8.36	12.95	81.50
TMR	8.92	22.06	33.37	25.00	9.44	22.92	32.21	26.00
MoPa	10.80	26.72	38.02	19.00	11.25	26.86	37.40	20.50

Table 3: In-distribution retrieval results (C2). MoCHA achieves state-of-the-art on both benchmarks, with consistent gains across all recall ranks and retrieval directions—ruling out a precision-recall tradeoff and confirming that the removed content was

a

, not useful

s

. Full ablations in Appendix 0.G.1 .

			T2M
Train	Model	Test	R@1	R@5	R@10
H3D	MoPa	KIT	13.74	37.79	53.31
	MoCHA (T5 blend-rev)	KIT	25.70	44.27	61.83
	MoCHA (LLM blend-rev)	KIT	26.59	48.35	61.07
KIT	MoPa	H3D	1.85	6.02	9.56
	MoCHA (T5 blend)	H3D	2.01	8.65	13.00

Table 4: Cross-dataset retrieval. Stripping

p(a)

removes dataset-specific annotation style, so models trained on one dataset’s conventions can retrieve from another’s. Gains are proportionally larger than in-distribution—consistent with the

(s,a)

decomposition in Section 3.1 , which predicts that

a

hurts most when the test distribution of

a

differs from training.


	T2M
	R@1	R@5	R@10
MoPa	10.80	26.72	38.02
$+$ Paraphrases	10.13 (-0.67)	27.62	40.31
MoCHA (T5)	13.96	31.41	43.82
MoCHA (LLM)	12.55	29.88	44.64

Table 5: Denoising the positive vs. augmenting it. Paraphrase augmentation widens

p(t\mid s)

rather than reducing it, acting as a smoothing function that trades R@1 for R@5/R@10. Canonicalization improves all ranks by collapsing variance rather than adding to it.

	Baseline	Backtrans	Stopword	FT5 (MoCHA)
H $\to$ H	10.80	8.71	10.58	13.12
K $\to$ K	14.02	11.45	15.39	20.74
H $\to$ K	13.74	15.39	16.67	18.96
K $\to$ H	1.78	1.71	2.17	2.78

Table 6: Canonicalization mechanism ablation (T2M R@1 %). Even rule-based stopword stripping improves transfer, while backtranslation (which transforms without denoising) does not—confirming that canonicalization is a general principle (C1): the gains stem from projecting onto

s

, not from any particular model’s language understanding.

Original	Backtrans	Stopword	FlanT5
“a person holding an item turns back, then left, then back, like he’s been followed ”	“a person holding an object turns back, then left, then back, as if you were following it ”	“holding item turns back left back he’s followed”	“hold item $\to$ turn back $\to$ turn left $\to$ turn back”
“a man holds something in both hands and walks in a counterclockwise oval, before coming to a stop, then resuming walking.”	“a man holds something in both hands and goes oval in counterclockwise sense before he comes to an attack , then goes again.”	“holds hands walks counterclockwise oval coming stop resuming walking”	“hold object both hands $\to$ walk counterclockwise oval $\to$ stop $\to$ walk”

Table 7: Example outputs by method. Backtranslation (negative control: transforms surface form without removing

a

) preserves nuisance factors intact (“as if you were following it”); stopword stripping removes noise indiscriminately, unlocking cross-dataset gains but losing temporal structure. MoCHA retains sequential actions and fine-grained

s

that rule-based methods cannot distinguish from

a

—illustrating why canonicalization is a general principle that even simple rules can exploit, but learned semantic projection yields the largest improvements.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	FlanT5-PPT	FlanT5-Rev
Base model	FlanT5-base (250M)	FlanT5-PPT ckpt
Training pairs	168K	$\sim$ 4.8K
Frozen layers	None	All exc. last 2 dec.
Learning rate	$3\times 10^{-4}$	$1\times 10^{-5}$
Batch size	128	32
Max epochs	10	5

Appendix

Table 8: FlanT5 [ 7 ] fine-tuning hyperparameters.

Hyperparameter	Value
Motion encoder	ViT-B/16, 22 joints
Text encoder	DistilBERT (CLS pool)
Embedding dim	256
Loss	Symmetric InfoNCE
Temperature $\tau$	0.07 (frozen)
Self-sim threshold	0.80 (mpnet)

Appendix

Table 9: MotionPatches (MoPa) [ 6 ] retrieval training hyperparameters.

Strategy	H3D	KIT
MoPa Baseline (orig text)	10.80	14.02
MoCHA Cardinal	12.66	23.66
MoCHA Blend	13.30	22.14
MoCHA Blend-Rev	12.14	20.74

Appendix

Table 10: MoCHA (T5) canonicalization strategy comparison (DsPair T2M R@1).

Model	H3D	KIT
TMR Baseline (orig)	8.96	11.70 †
TMR Pure Cardinal	10.31	19.08
TMR Blend	13.64	17.05
TMR Blend-Rev	11.09	17.68

Appendix

Table 11: Canonicalization on TMR (DsPair T2M R@1, canonical text at test). Evaluated at best validation epoch. † KIT baseline corrected after fixing case-sensitivity in mirror-caption grouping.

HumanML3D native
Config	s42	s123	s456	Mean $\pm$ Std
Baseline (orig)	11.72	11.18	10.69	11.20 $\pm$ 0.52
MoCHA (LLM) Blend-Rev	14.09	11.72	13.14	12.98 $\pm$ 1.19
MoCHA (T5) Blend-Rev	13.73	12.24	12.20	12.72 $\pm$ 0.87
KIT-ML native
Baseline (orig)	14.31 †	11.58 †	13.87 †	13.25 $\pm$ 1.20 †

Appendix

Table 12: Retrieval performance across 3 random seeds (DsPair Avg R@1). Each model evaluated at its own best validation epoch. “+canon” = canonical text at test time. † KIT baseline re-evaluated after correcting case-sensitivity in mirror-caption grouping.

			T2M
Train	Model	Test	R@1	R@5	R@10
H3D	MoPa	BABEL	15.84	43.32	49.31
	MoCHA (T5 blend-rev)	BABEL	22.36	35.57	42.07
	MoCHA (LLM blend)	BABEL	26.09	46.76	55.92
KIT	MoPa	BABEL	12.76	29.93	36.99
	MoCHA (T5 blend-rev)	BABEL	25.11	29.66	30.90

Appendix

Table 13: BABEL cross-dataset retrieval (Threshold T2M protocol per TMR++). X

\to

B comparisons use last epoch as BABEL R@1 is unstable due to caption duplicity. BABEL numbers are inflated by 86% caption duplication (see text). H

\to

K and K

\to

H results without BABEL are reported in the main paper (Table 4).

	H3D			KIT
Train	Orig	LLM	T5	Orig	LLM	T5
MoPa Baseline	11.13	7.30	9.24	8.91	15.39	13.87
MoCHA Cardinal	6.16	12.52	11.93	6.36	23.92	23.03
MoCHA Blend	11.82	13.21	12.18	10.69	15.39	18.32
MoCHA Blend-Rev	11.45	13.91	14.03	8.65	24.30	25.83

Appendix

Table 14: Test-time text mode ablation (DsPair T2M R@1, epoch 50). Each row is a different training strategy (LLM-trained models); columns show performance under each test-time text mode. Note: Main Table 3 MoCHA (T5) reports a separately-trained FlanT5-PPT Blend model (13.30%/22.14%); see Appendix 0.G.1 for all FlanT5-PPT variants.

HumanML3D (4,382 multi-caption motions)
	Condition	Intra	Align	Inter NN	Sep
	Baseline + orig	0.413	0.601	0.976	0.423
	Blend-Rev + orig	0.444	0.634	0.970	0.458
	Blend-Rev + canon	0.444	0.636	0.971	0.457
KIT-ML (393 multi-caption motions)
	Baseline + orig	0.456	0.515	0.941	0.484

Appendix

Table 15: Full embedding space geometry in the trained encoder’s 256-dim retrieval space. Intra: text-text similarity within same-motion captions. Align: text-motion cosine similarity. Inter NN: nearest negative similarity. Sep: Intra / Inter NN (higher = better separation).

Data	Bin	N	Base	Blend-Rev	$\Delta$
H3D	short (1–6)	1,145	17.78	17.71	$-$ 0.07
H3D	medium (7–12)	2,382	12.97	16.73	+3.76
H3D	long (13+)	856	12.07	12.68	+0.61
KIT	short (1–6)	523	13.21	21.65	+8.44
KIT	medium (7–12)	161	19.52	32.11	+12.59
KIT	long (13+)	102	21.22	17.65	$-$ 3.57

Appendix

Table 16: Caption length vs. R@1 gain (DsPair T2M R@1).

\Delta

: Blend-Rev R@1 minus baseline R@1 (percentage points).

	HumanML3D		KIT-ML
Model	Entropy	P(+)	Entropy	P(+)
MoPa	6.285	0.0158	4.939	0.0300
$+$ Paraphrases	6.093	0.0156	4.901	0.0288
MoCHA (blend-rev)	6.034	0.0190	4.673	0.0337

Appendix

Table 17: InfoNCE gradient concentration. Softmax entropy and P(positive) measured on trained models’ test-set similarity matrices. Lower entropy and higher P(positive) indicate more concentrated gradient signal toward the correct positive.

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

Aug 8, 2026

MRBench: A Comprehensive Benchmark for Human Motion-Text Retrieval

Human motion-text retrieval provides a rigorous means of assessing cross-modal alignment. Prevailing benchmarks are dominated by homogeneous indoor motions, imbalanced motion distributions, and oversimplified, repetitive texts, which hinder the reliable measurement of cross-domain and cross-granularity alignment. We thus introduce MRBench, a comprehensive motion-text retrieval benchmark featuring heterogeneous motions, broad and balanced category coverage, and reliable, discriminative, multi-granular descriptions. MRBench is constructed through a meticulously designed multi-stage data curation pipeline, which filters and balances candidates, verifies unambiguous semantic alignment, and generates motion-grounded descriptions at multiple granularities. The resulting benchmark contains 3,390 motions drawn from motion capture, in-the-wild videos, synthetic videos, and motion generative models, covering 118 fine-grained categories. Each motion is paired with concise, standard, and fine-grained descriptions, yielding 10,170 captions. Extensive evaluations of representative retrieval baselines on MRBench reveal a substantial cross-dataset generalization gap and pronounced sensitivity to query granularity. We propose a lightweight granularity-aware model anchored at a frozen standard-caption-aligned retrieval model. LLM-based concise and fine-grained captions provide pseudo-supervision for extra-branch granularity-specific motion extractors and text adapters. For inference, granularity-aware score fusion integrates global and adapted similarities while strictly maintaining score comparability across all description levels. The resulting model improves mixed-granularity retrieval without compromising standard-caption performance. We believe that our MRBench provides a comprehensive testbed for advancing motion-language alignment evaluation.

Fulong Liu, Liang Xu, Chengqun Yang +3

Shanghai Jiao Tong University · Beijing Zhongguancun Academy

Cross-Modal Alignment Information Retrieval Vision-Language Retrieval

Ask PDF

Mar 10, 2026

Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction

Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.

Yao Zhang, Zhuchenyang Liu, Yanlan He +2

Aalto University, Espoo, Finland · Fudan University, Shanghai, China · Georgia Institute of Technology, Atlanta GA, USA

Representation Learning Feature Interaction Modeling Cross-Modal Retrieval

Ask PDF

Aug 2, 2026

FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision

Text-conditioned human motion generation has made rapid progress with the emergence of large-scale motion--language datasets. However, even datasets with rich long-form descriptions typically provide supervision only at the clip level, without explicit temporal correspondence between motion frames and language. This limits fine-grained motion--text grounding and temporally precise generation. We propose FineMoLA, a weakly supervised framework that learns fine-grained frame--phrase correspondence directly from clip-level annotations. Our method first segments long-form descriptions into action-bearing phrases, and then formulates motion--language alignment as an optimal transport problem, which naturally models many-to-many relations between motion frames and text under global constraints. With entropic regularization and Sinkhorn iterations, FineMoLA efficiently infers pseudo frame-level alignments without human labeling. Experiments on SnapMoGen demonstrate that the learned alignments outperform baselines in motion--text grounding.

Tongyan Wang, Zhengyuan Li, Muhan Lin +5

1Purdue University · University of Illinois Urbana-Champaign

Cross-Modal Alignment

Ask PDF

Abstract

Cross-Modal Retrieval

Explore similar work

MRBench: A Comprehensive Benchmark for Human Motion-Text Retrieval

Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction

FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision