Traceable Human-to-Humanoid Sign Language Benchmarking

Ao Liu; Shengeng Tang; Lechao Cheng; Yanbin Hao; Bingkun Bao; Richang Hong

cs.ROSep 27, 2026

Traceable Human-to-Humanoid Sign Language Benchmarking

Authors: Ao Liu, Shengeng Tang, Lechao Cheng, Yanbin Hao, Bingkun Bao, Richang Hong

Organizations: Hefei University of Technology

Abstract

Sign data collection is costly, and teleoperation scales poorly, motivating reuse of large video corpora. Humanoid signing requires converting video-derived human motion into robot trajectories while preserving linguistic motion cues. Errors from fitting, human-motion repair, retargeting, robot geometry repair, and control are hard to separate from the final trajectory alone. We introduce HumanoidCSL-20K, a dataset and benchmark of 20,648 sentence-level Chinese Sign Language sequences, each with four aligned versions: the source, the repaired human motion, the direct robot reference, and the geometry-repaired robot reference. Observation-supported local human-motion repair, full-robot geometry repair, and cross-representation provenance make each transformation traceable. Paired evaluations measure human-motion continuity and content preservation, robot-reference feasibility, and physical execution. A sign-specific kinematic-reference protocol scores handshape, location, palm orientation, and inter-hand relation over the full planned motion. Full-corpus results show fewer abnormal arm / hand steps and less inter-hand and hand-body penetration after repair. Control experiments separate reference learnability from curriculum effects, while component scores expose remaining execution errors.

Figures & tables

Figure 1: HumanoidCSL-20K links 20,648 signing sequences across four aligned versions of human and robot motion. Simulated examples and benchmark results show fewer abrupt joint changes and robot self-collisions after repair, plus higher control scores under the same training budget.

Figure 2: Why fitted human motion is not directly robot-ready. (a) Video fits contain local jumps and frozen hands. (b) Plausible human motion penetrates the robot’s own body after retargeting. (c) A collision-free robot reference can still fail in execution. Replay stops at the first failure, and unreached frames score zero.

Figure 3: Pipeline overview: (a) linked human and robot motion construction; (b) body-wrist tracking followed by fine handshape tracking.

Figure 4: Source-motion diagnostics over 20,652 sequences: (a) rotation-step distributions, (b) affected-frame rates, (c) short motion bursts, and (d) long static-hand spans.

Figure 5: Representative local repair of isolated (a) wrist and (b) handshape discontinuities while retaining neighboring anchors. Boxes mark corrections, and leaders connect poses to curves.

Panel of Figure 6 — Figure 6: Geometry repair examples for hand-head, hand-leg, inter-hand, and intra-hand penetration; red marks collisions.

	AMP			Ours, single-stage			Ours, two-stage + ramp
Reference	AS $\uparrow$	Fin. $\uparrow$	Pen. $\downarrow$	AS $\uparrow$	Fin. $\uparrow$	Pen. $\downarrow$	AS $\uparrow$	Fin. $\uparrow$	Pen. $\downarrow$
Raw + Direct	2.73	0.0	32.24	34.99	56.7	36.15	35.51	60.0	35.53
Fused + Direct	2.68	0.0	26.39	49.15	96.7	25.97	49.29	98.3	26.08
Fused + Geometry	2.79	0.0	23.50	52.58	100.0	16.09	54.10	100.0	13.03

Table 3: Execution on 20 motions and three seeds. Reading down, reference repair improves both of our controllers; reading across, our controllers far exceed AMP, and the full curriculum is best on the repaired reference.

Figure 7: Replay on a physical Unitree G1 humanoid: source-signer frames (top) and the corresponding robot replay (bottom).

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

State	Motion payload	Linked records
$\mathcal{H}^{\mathrm{raw}}$	SMPL-X root, translation, shape, body, and hand arrays	Video and sentence IDs, source frames and timestamps, coordinate convention
$\mathcal{H}^{\mathrm{fused}}$	Same human-motion schema	Part masks, repair intervals, anchors, operations, observation-support scores
$\mathcal{G}^{\mathrm{direct}}$	Floating base and 53-coordinate robot motion	Robot model, joint ordering, collision model, coupling definitions
$\mathcal{G}^{\mathrm{repair}}$	Same robot-motion schema	Deformation, residual penetration, constraint checks, certification flag

Appendix

Table 4: Stage-wise motion representations and linked records.

Provenance category	Score
Retained source fit	1.00
Exit transition from a frozen segment	0.70
Two-anchor interpolation	0.50
Single-anchor freezing	0.25
Long-segment freezing	0.10

Appendix

Table 5: Operation labels and fixed scores for human-motion repair provenance.

Set	Size	Source	Role
$\mathcal{D}_{\mathrm{all}}$	20,652	Full source manifest	Production coverage, corpus statistics, and paired human and geometry evaluation
$\mathcal{D}_{\mathrm{diag}}$	600	Fixed stratified subset	Repair-cue diagnostics (Figure 8 )
$\mathcal{D}_{\mathrm{content}}$	75	Recognizer-held-out dev/test sequences	Content preservation (frozen recognizer)
$\mathcal{D}_{\mathrm{control}}$	20	Fixed random sample	Self-target and fixed-reference replay

Appendix

Table 6: Data partitions used by the construction and evaluation pipeline.

Sequence ID	Sentence-level meaning (English translation)
S000374_P0004_T00	I go to bed early too, at ten every night. Going to bed early is good for your health.
S000443_P0000_T00	Take a break.
S000748_P0004_T00	Take the bus to the train station from across the road.
S000919_P0008_T00	What tasty dishes does your restaurant have? Please recommend some.
S001518_P0004_T00	Why do you drink coffee at night?
S001836_P0000_T00	Only by delighting the eye and mind can works of art move and educate people.

Appendix

Table 7: The 20 control motions with English translations of their CSL-Daily sentence annotations.

Figure 8: Image-cue rates on frames flagged by the diagnostic motion rule (red) and on unflagged frames (blue) in the fixed 600-sequence diagnostic set.

Group	DoF	Joints and ranges
Legs	12	Hip pitch, roll, yaw; knee; ankle pitch, roll (per leg)
Waist	3	Yaw, roll, pitch
Arms	14	Shoulder pitch, roll, yaw; elbow; wrist roll, pitch, yaw (per arm)
Hands, actuated	12	Thumb rotation $[0,1.164]$ rad; thumb flexion $[0,0.586]$ rad; index, middle, ring, and little flexion $[0,1.438]$ rad
Hands, passive	12	Distal thumb and finger joints coupled to the actuated joints
Total	53	41 actuated coordinates, 12 mimic constraints

Appendix

Table 8: Kinematic structure of the G1 + RH56DFTP platform. Hand ranges are given per hand.

Set	Category	Pairs
$\mathcal{K}_{\mathrm{body}}$	Inter-hand	289
	Hand-contralateral arm	136
	Hand-same-side arm	134
	Hand-torso	102
	Hand-head	34
	Hand-leg	612

Appendix

Table 9: Audited collision pairs of the G1 + RH56DFTP model.

Figure 9: Unitree G1 with Inspire RH56DFTP hands. (a) Visual model. (b) Collision proxies colored by group. (c) Hand detail with the twelve finger-segment capsules; alternating shades distinguish adjacent segments. All views are rendered from the production MuJoCo model.

Quantity	Value	Implementation value	Role
$\lambda_{r}$	$144$	$w_{r}=12$	Inter-wrist displacement residual
$\lambda_{v}$	$0.3$	$0.3$	First joint difference
$\lambda_{a}$	$8$	$8$	Second joint difference
$\lambda_{j}$	$100$	$100$	Third joint difference
$\lambda_{c}$	$1$	$1$	Wrist-residual continuity (Equation 14 )
$\lambda_{0}$	$0.02$	$0.02$	Deviation from the projected initialization

Appendix

Table 10: Geometry-repair objective and metric scalings. Squared implementation weights are shown as their effective coefficients in Equation 15 .

Parameter	Value	Unit / Dimension	Role
$\theta_{\max}$	$20.0^{\circ}$ ( $0.34907$ )	rad	Hard per-joint budget on active finger joints
$\gamma_{f}$	$0.2$	mm ( $2\times 10^{-4}$ m)	Target geometric clearance margin
$\beta_{1}$	$0.5$	dimensionless	First difference regularizer ( $D_{1}$ )
$\beta_{2}$	$5.0$	dimensionless	Second difference regularizer ( $D_{2}$ )
$\beta_{3}$	$25.0$	dimensionless	Third difference regularizer ( $D_{3}$ )
$\rho$	$10{,}000$	$\rho^{2}=10^{8}$	Quadratic penalty on penetration slack $s_{k,\tau}$

Appendix

Table 11: Implementation parameters for the finger stage (Equation 16 ).

Term	Unweighted term $r_{k}$	$\sigma$	$w_{k}^{(1)}$	$w_{k}^{(2)}$
Anchor position	$r_{\mathrm{trk}}(\lVert\bm{p}-\bm{p}^{\mathrm{ref}}\rVert_{2}^{2};\sigma)$	0.50 m	1.00	1.00
Anchor orientation	$r_{\mathrm{trk}}(d_{SO(3)}^{2};\sigma)$	0.60 rad	0.75	0.75
Body position	$r_{\mathrm{trk}}(E_{\mathrm{vec}};\sigma)$ , 14 bodies	0.35 m	3.00	3.00
Body orientation	$r_{\mathrm{trk}}(E_{\mathrm{rot}};\sigma)$ , 14 bodies	0.60 rad	1.00	1.00
Body linear velocity	$r_{\mathrm{trk}}(E_{\mathrm{vec}};\sigma)$ , 14 bodies	1.00 m/s	0.25	0.25
Body angular velocity	$r_{\mathrm{trk}}(E_{\mathrm{vec}};\sigma)$ , 14 bodies	3.14 rad/s	0.25	0.25

Appendix

Table 12: Reward terms used by the reported Stage-1 and Stage-2 tasks. Tracking rows use

r_{\mathrm{trk}}

from Equation 18 . A dash denotes an inactive term. The contact term counts configured bodies whose peak force in the sensor history exceeds 1 N.

Observation block	Actor	Critic
Reference joint positions and velocities	106	106
Reference-anchor relative position	–	3
Reference-anchor relative orientation	6	6
Tracked-body positions ( $14\times 3$ )	–	42
Tracked-body orientations ( $14\times 6$ )	–	84
Base linear velocity	–	3

Appendix

Table 13: Actor and critic observation spaces. A dash denotes a block absent from that input. Reference position and velocity are the current 53-coordinate targets on the 50 Hz control grid.

Metric	Fused $-$ Raw	Fused $-$ Global
Arm abnormal steps (pp)	$-18.725\;[-18.823,-18.627]$	$-1.143\;[-1.175,-1.111]$
Hand abnormal steps (pp)	$-10.554\;[-10.619,-10.488]$	$-0.140\;[-0.154,-0.126]$

Appendix

Table 14: Paired continuity effects over 20,649 sequences. Entries are mean differences with 95% sequence-bootstrap intervals; pp denotes percentage points.

Group / condition	Changed units (%)	Path ratio (%)	Step variation	Source dev. (rad)
Arms / Raw	0.00	100.00	5618.9	0.0000
Arms / Global smoothing	99.68	47.21	585.4	0.1391
Arms / Fused repair	79.75	36.49	1037.3	0.3741
Hands / Raw	0.00	100.00	7495.1	0.0000
Hands / Global smoothing	85.17	58.95	2722.4	0.1134
Hands / Fused repair	61.34	34.23	1371.9	0.3807

Appendix

Table 15: Supplementary motion diagnostics on all 20,649 pairs. Path ratio is rotation-path length relative to Raw; source deviation is measured from Raw.

Group / condition	Supported dev. (rad)	Supported changed (%)	Flagged dev. (rad)	From flagged (%)
Arms / Global smoothing	0.0769	99.62	0.1632	82.5
Arms / Fused repair	0.0418	38.94	0.5246	96.4
Hands / Global smoothing	0.0594	99.57	0.1364	83.5
Hands / Fused repair	0.0000	0.00	0.5549	100.0

Appendix

Table 16: Source deviation from Raw split by the stored repair mask. Supported side-frames lie outside the mask (32.7% of side-frames) and flagged side-frames inside it (67.3%). Deviation means and changed shares pool side-frames over all 20,649 sequences; the last column is the share of the sequence-averaged source deviation in Table 15 that comes from flagged side-frames.

Human motion	BLEU-1	BLEU-2	BLEU-3	BLEU-4
Raw	54.39	41.46	32.88	26.59
Global smoothing	48.65	37.33	30.06	24.50
Fused repair	54.10	41.17	32.64	26.29

Appendix

Table 17: All BLEU orders for Raw, Global smoothing, and Fused repair on the same 75-sequence content cohort.

Contrast / metric	Difference [95% interval]	Four-order interval
Global $-$ Raw / BLEU-1	$-5.735\;[-8.869,-2.633]$	$[-9.766,-1.736]$
Global $-$ Raw / BLEU-2	$-4.129\;[-7.056,-1.249]$	$[-7.973,-0.441]$
Global $-$ Raw / BLEU-3	$-2.825\;[-5.592,-0.080]$	$[-6.460,0.656]$
Global $-$ Raw / BLEU-4	$-2.092\;[-4.713,0.532]$	$[-5.430,1.167]$
Fused $-$ Raw / BLEU-1	$-0.290\;[-1.420,0.911]$	$[-1.782,1.240]$
Fused $-$ Raw / BLEU-2	$-0.290\;[-1.781,1.199]$	$[-2.254,1.704]$

Appendix

Table 18: Paired content effects across all BLEU orders. The final column adjusts for four orders within each contrast.

Metric	Global $-$ Raw (Direct)	Geometry $-$ Fused + Direct	Geometry $-$ Global + Direct
Pen. frames (pp)	$-0.637$ $[-0.706,-0.570]$	$-49.893$ $[-50.204,-49.576]$	$-52.182$ $[-52.434,-51.925]$
Inter-frame pen. (pp)	$-0.663$ $[-0.730,-0.597]$	$-49.902$ $[-50.213,-49.585]$	$-52.195$ $[-52.448,-51.936]$
Mean pen. depth (mm)	$-0.383$ $[-0.419,-0.347]$	$-8.270$ $[-8.357,-8.184]$	$-10.991$ $[-11.081,-10.903]$
Pen.-free clips (pp)	$0.140$ $[0.082,0.199]$	$87.180$ $[86.720,87.631]$	$88.178$ $[87.742,88.624]$

Appendix

Table 19: Paired effects for the four-condition geometry comparison over all 1,519 pairs in

\mathcal{K}_{\mathrm{eval}}

and 20,648 matched sequences. Entries give the difference and 95% interval; pp denotes percentage points. Geometry denotes Fused + Geometry.

	Grid frames (%)		Interior samples (%)
Pair scope	Direct	Geometry ( $\mathcal{G}^{\mathrm{repair}}$ )	Direct	Geometry ( $\mathcal{G}^{\mathrm{repair}}$ )
Inter-hand + hand-body	26.6152	0.0993	26.6520	0.1140
Intra-hand	31.4145	0.2841	31.4028	0.2848
All audited pairs	50.1835	0.2904	50.2044	0.3027

Appendix

Table 20: Per-category penetration for Fused + Direct and Fused + Geometry (

\mathcal{G}^{\mathrm{repair}}

) at 50 Hz across all 20,648 sequences. All entries are macro percentages of penetrating samples. Interior samples use three fixed points per resampled interval.

Robot reference	Output	Checks passed	Certified	Joint RMS (rad)	Relation RMS (cm)	Runtime (s)
Fused + Direct	20,649	0	0	0.0000	0.000	2.46
Fused + Geometry	20,648	20,648	18,214	0.1886	7.955	366.93

Appendix

Table 21: Construction outcome for the Fused production pair. Counts use all 20,652 planned inputs; deformation and runtime average over the 20,648 completed pairs. RMS measures the change from Direct, and runtime is concurrent CPU wall-clock time per sequence.

Group	Sequences	At $20^{\circ}$ cap	Penetrating samples (%)
(i) Control-grid only	762	52	0.18
(ii) Native intra-hand	615	562	3.03
(iii) Native cross-part	1,057	810	3.19
All uncertified	2,434	1,424	2.21

Appendix

Table 22: Partition of the 2,434 uncertified sequences by failed validation check. Groups are mutually exclusive. Penetrating samples pool the 50 Hz grid and interior samples over all audited pairs.

Component	Per-frame error	Nominal	Alternative
Handshape	Largest absolute error over 12 active joints	0.30 rad	0.50 rad
Location	Larger left/right wrist position error	10 cm	15 cm
Orientation	Larger left/right palm rotation error	$0.50$ rad	$0.80$ rad
Relation	Error in the right-minus-left wrist vector	10 cm	15 cm

Appendix

Table 23: Nominal and alternative tolerance profiles. The palm tolerances are about

28.6^{\circ}

and

45.8^{\circ}

Controller / reference	ActionScore $\uparrow$	JointPass (%) $\uparrow$	Finished (%) $\uparrow$	Pen. (%) $\downarrow$	Depth (mm) $\downarrow$
(a) Ours, single-stage
Raw + Direct	34.99	7.50	56.7	36.15	0.548
Global + Direct	50.17	12.46	100.0	27.67	0.309
Fused + Direct	49.15	13.56	96.7	25.97	0.865
Fused + Geometry	52.58	16.32	100.0	16.09	0.138
(b) Ours, two-stage + ramp

Appendix

Table 24: Complete controlled execution results on 20 motions and three seeds. Within each group, rows differ only in the reference; the Fused + Geometry rows of (a)–(c) differ only in the curriculum. Shading marks our full pipeline, and bold marks the best value among our controllers.

	ActionScore $\uparrow$			Finished (%) $\uparrow$
Reference	42	1234	777	42	1234	777
(a) Single-stage training
Raw + Direct	30.00	36.93	38.05	45	60	65
Global + Direct	49.27	50.30	50.92	100	100	100
Fused + Direct	50.13	49.46	47.85	100	100	90
Fused + Geometry	52.20	52.03	53.51	100	100	100

Appendix

Table 25: Per-seed control results on the same 20 motions. The main table averages seeds within each motion before averaging motions. Finished measures replay completion; ActionScore also accounts for reached-frame fidelity.

	Coverage	Reached-frame RMS $\downarrow$			Pen. $>1$ mm
Reference	(%) $\uparrow$	Hand (rad)	Wrist (cm)	Palm ( ∘ )	(%) $\downarrow$
(a) Single-stage training: reference processing
Raw + Direct	68.8	0.0896	13.02	48.50	15.69
Global + Direct	100.0	0.0821	14.45	43.65	13.08
Fused + Direct	98.4	0.0926	14.55	43.81	10.51
Fused + Geometry	100.0	0.0792	13.97	40.97	5.51

Appendix

Table 26: Reached-frame tracking errors and deeper penetration for the same trials as Table 24 . Coverage exposes the interval on which conditional errors are measured. The fixed Fused + Geometry curriculum comparison has complete coverage in every condition.

	Replay		Additional diagnostics $\downarrow$			Depth (mm) $\downarrow$
Reference	Finished (%) $\uparrow$	Coverage (%) $\uparrow$	Relation (cm)	Limit (%)	Coupling (rad)	Mean	Peak
(a) Single-stage training: reference processing
Raw + Direct	56.7	68.8	13.37	0.476	0.0957	0.548	9.21
Global + Direct	100.0	100.0	12.32	0.274	0.0629	0.309	6.33
Fused + Direct	96.7	98.4	11.37	0.307	0.0687	0.865	10.65
Fused + Geometry	100.0	100.0	10.38	0.498	0.0614	0.138	3.29

Appendix

Table 27: Replay stability and physical diagnostics for the same trials as Table 24 . Means average seeds within each motion and give equal weight to each motion. Depth statistics use the positive depth of the worst audited pair at each executed frame, with zero for separated pairs.

	Continuous component score				ActionScore		CA
Reference	Hand- shape	Location	Orient.	Relation	Nominal	Alt.	CA nom. (%)
(a) Single-stage training: reference processing
Raw + Direct	53.55	30.56	22.80	33.05	34.99	47.84	35.94
Global + Direct	81.24	36.11	35.30	48.03	50.17	69.32	50.06
Fused + Direct	77.30	34.50	33.02	51.77	49.15	68.05	48.44
Fused + Geometry	81.34	36.37	36.48	56.13	52.58	71.75	51.78

Appendix

Table 28: ActionScore decomposition and tolerance sensitivity. The first four columns report component scores under the nominal scales using the complete planned-frame denominator; their mean gives the nominal ActionScore. Alternative-scale ActionScore applies the alternative tolerance profile to the same recorded trajectories. CA reports component-wise hard-threshold attainment under the nominal profile. All columns are higher-is-better.

Reference	Hand- shape	Location	Orient.	Relation	CA nominal	CA alternative	JointPass alternative
(a) Single-stage training: reference processing
Raw + Direct	61.26	29.67	18.30	34.53	35.94	54.12	23.47
Global + Direct	92.62	29.89	29.66	48.07	50.06	76.50	37.67
Fused + Direct	86.40	27.77	25.63	53.94	48.44	75.47	41.26
Fused + Geometry	91.05	28.89	28.96	58.21	51.78	80.35	49.95
(b) Two-stage training: reference processing and ramp ablation

Appendix

Table 29: Hard-threshold execution metrics under the nominal and alternative tolerance profiles. The first four columns report component-wise attainment over all planned frames under the nominal profile, and their average gives nominal CA. Alternative CA and JointPass use the alternative profile. These quantities are percentages and are distinct from the continuous ActionScore reported in Table 24 .

Reference	Windows (%) $\uparrow$	Hand	Wrist	Palm	Relation
Velocity error (hand and palm in rad/s; wrist and relation in m/s)
(a) Single-stage training: reference processing
Raw + Direct	68.6	1.042	1.212	11.546	1.324
Global + Direct	100.0	0.741	0.721	5.090	0.816
Fused + Direct	98.4	0.657	0.592	4.489	0.637
Fused + Geometry	100.0	0.517	0.556	4.072	0.616

Appendix

Table 30: Velocity and acceleration tracking errors relative to the reference. RMS is computed on reached windows per trial, averaged across seeds within each motion and then equally across motions. Window coverage uses the planned denominator. All error columns are lower-is-better.

Panel	Condition	CA area	JointPass area
a	Raw + Direct	47.454	21.671
a	Global + Direct	69.305	37.678
a	Fused + Direct	68.044	39.974
a	Fused + Geometry	71.755	46.943
b	Raw + Direct	49.029	22.492
b	Fused + Direct	68.522	39.803

Appendix

Table 31: Normalized threshold areas on

[0.25,3]

, on a 0–100 scale. Each condition contains 20 motions and three seeds. The panel identifiers match Figure 10 .

Figure 10: CA and JointPass under common scaling of all four action thresholds. Rows (a,b) compare references under single- and two-stage training, (c) compares the three curricula on Fused + Geometry, and (d) shows AMP (Table 24 ). Dotted lines mark the nominal scale. AMP uses a separate vertical scale. Curves are drawn on a dense display grid; areas and ordering checks use the exact empirical breakpoints.

		Right side			Left side
Operation	Score	Share (%)	Arm RMS (rad)	Hand RMS (rad)	Share (%)	Arm RMS (rad)	Hand RMS (rad)
Retained source fit	1.00	65.43	0.162	0.062	36.08	0.154	0.057
Exit transition	0.70	0.12	0.129	0.054	2.00	0.173	0.059
Two-anchor interpolation	0.50	19.07	0.179	0.070	6.23	0.181	0.070
Single-anchor freezing	0.25	3.52	0.150	0.105	2.16	0.148	0.057
Long-segment freezing	0.10	0.50	0.240	0.061	42.17	0.105	0.065

Appendix

Table 32: Tracking error of the multi-motion policy grouped by reference provenance. Share is the percentage of frames of that side; RMS pools joint-frames and averages the three seeds.

Figure 11: Human fits and robot poses before and after geometry repair. Red links and crosses mark penetration; checks indicate its removal.

Figure 12: Source signing and hardware replay for four utterances. Each row contains five chronological frame pairs, with the source signer on the left and the real robot on the right. Row labels summarize the annotated utterance meanings. Pairs use the global motion-phase alignment described above; fixed crops retain the upper body and hands while excluding wall lettering.

Figure 13: Short-gap interpolation reduces the local hand-rotation step peak from 1.01 to 0.15 rad.

Figure 14: Holding and exit blending stabilize a partly out-of-frame hand, reducing the local rotation-step peak from 1.21 to 0.33 rad.

Figure 15: Handshape holding and exit blending bridge an extended interval, reducing the local rotation-step peak from 1.45 to 0.29 rad.

Figure 16: Repair and local smoothing suppress left-wrist jitter, reducing the displayed-window displacement peak from 317.4 to 10.5 mm/frame.

Figure 17: Holding a representative low-arm pose suppresses wrist excursions toward the torso, reducing the displacement peak from 583.8 to 17.2 mm/frame.

Figure 18: A left-arm hold suppresses wrist jitter while the right arm signs, reducing the displacement peak from 178.5 to 24.5 mm/frame.

Figure 19: Holding a lowered arm reduces peak wrist displacement from 190.7 to 17.1 mm/frame.

Explore similar work

Jul 20, 2026cs.RO

From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation

Recent sign language generation (SLG) systems increasingly output dense 3D body representations, which better preserve full-body kinematics and geometry for downstream embodiment on humanoid robots. However, these generated motions frequently exhibit self-intersections such as hand-hand and hand-torso penetration. While such artifacts may be tolerated in offline rendering, they become critical in humanoid execution as they lead to infeasible inverse-kinematics (IK) solutions, collisions, and unstable retargeted trajectories. We present a system-level framework that bridges SLG outputs to humanoid joint-space execution via two components. First, we introduce a volumetric SMPL-X collision-mitigation module that projects generated signing motions toward physically plausible configurations while minimally deviating from the original trajectory. Second, we propose a vision-language-guided retargeting algorithm built on an IK backbone: a VLM serves as a visual critic over rendered humanoid motion, identifies embodiment-specific failure modes, and triggers targeted task-space corrections. Our results highlight collision handling and perception-guided refinement as key missing components for reliable humanoid signing.

Nabeela Khan, Bowen Wu, Runwu Shi +5

Department of Systems and Control Engineering, Institute of Science Tokyo, Tokyo, Japan · RIKEN Guardian Robot Project, Kyoto, Japan · ATR Hiroshi Ishiguro Laboratories, Kyoto, Japan +1

Sign Language Translation Cross-Embodiment Retargeting Embodiment Collision Prediction

Ask PDF

Jun 1, 2026cs.CV

Conditional Collapse in Sign Language Production: A Diagnostic and a Scaling Argument

Sign Language Production (SLP) is the task of generating avatar sign language motion from natural language text. The quality of the generated motion is typically evaluated by a motion-space Fréchet distance (FID) and back-translation (BT) BLEU score on benchmarks such as How2Sign. Both metrics can improve substantially while the underlying generator fails to faithfully represent the sign language gestures. In this work we propose to evaluate the generated motion at three independent levels: (

\tau1

) initial-pose conditioning, (

\tau2

) output diversity, and (

\tau3

) target faithfulness. We compute these as pairwise-distance ratios using latent representations of a frozen motion autoencoder (MoAE). We evaluate 14 SLP model checkpoints on the How2Sign dataset, including a re-implemented Neural Sign Actors (NSA), and show that

\tau3

faithfulness is never attained, while FID varies by nearly two orders of magnitude and is uncorrelated with faithfulness. We show that on the isolated gloss dataset ASL3DWord favorable

\tau3

can be attained, hence isolating the size of the sentence-level paired-dataset as the bottleneck.

Rui Hong, Jana Košecká

George Mason University

Sign Language Translation Speech Language Models Collapse Sequential Scaling

Ask PDF

May 3, 2026cs.CV

SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languages

Existing large-scale sign language resources typically provide supervision only at the level of raw video-text alignment and are often produced in laboratory settings. While such resources are important for semantic understanding, they do not directly provide a unified interface for open-world recognition and translation, or for modern pose-driven sign language video generation frameworks: 1. RGB-based pretrained recognition models depend heavily on fixed backgrounds or clothing conditions during recording, and are less robust in open-world settings than style-agnostic pose-processing models. 2. Recent pose-guided image/video generation models mostly use a unified keypoint representation such as DWPose as their control interface. At present, the sign language field still lacks a data resource that can directly interface with this modern pose-native paradigm while also targeting real-world open scenarios. We present SignVerse-2M, a large-scale multilingual pose-native dataset for sign language pose modeling and evaluation. Built from publicly available multilingual sign language video resources, it applies DWPose in a unified preprocessing pipeline to convert raw videos into 2D pose sequences that can be used directly for modeling, resulting in a consolidated corpus of about two million clips covering more than 55 sign languages. Unlike many laboratory datasets, this resource preserves the recording conditions and speaker diversity of real-world videos while reducing appearance variation through a unified pose representation. Toward this goal, we further provide the data construction pipeline, task definitions, and a simple SignDW Transformer baseline, demonstrating the feasibility of this resource for multilingual pose-space modeling and its compatibility with modern pose-driven pipelines, while discussing the evaluation claims it can support as well as its current limitations.

Sen Fang, Hongbin Zhong, Yanxin Zhang +1

1Rutgers University · 2Georgia Institute of Technology · 3NVIDIA

Sign Language Translation Sign Video Dataset Multimodal Dataset Transformer Architectures

Ask PDF