KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

Shuangyu Xie; Kaiyuan Chen; Muhammad Zubair Irshad; Richard Cheng; Sergey Zakharov; Ken Goldberg

cs.ROOct 22, 2026

KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

Authors: Shuangyu Xie, Kaiyuan Chen, Muhammad Zubair Irshad, Richard Cheng, Sergey Zakharov, Ken Goldberg

Organizations: University of California, Berkeley · Toyota Research Institute

Abstract

Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: https://berkeleyautomation.github.io/kirose/)

Figures & tables

Figure 1: Encoding Kinematic Events into Sparse Motion Tokens. Across diverse manipulation tasks and objects, continuous robot motion decomposes into recurring, temporally localized patterns termed kinematic events (e.g., bimanual lift, handover, pick, place). Despite visual and spatial variations, end-effector trajectories within the same motion primitive share structural kinematic signatures. KiRoSe compresses kinematic events into a short, ordered sequence of $N$ discrete Sparse Motion Tokens (colored blocks; shaded chronologically).

Figure 2: KiRoSe Pipeline. Given a robot kinematic stream and the task instruction “fold the paper box,” KiRoSe encodes motion using a transition-supervised encoder, then temporally pools and quantizes its features into Sparse Motion Tokens with a learned codebook. The language model reads these tokens, time markers, and the instruction to output action descriptions with start and end times, such as “pick up the object,” “place in the box,” and “adjust.” Transition annotations provide training supervision, while the motion tokenizer remains frozen during language-model adaptation.

Table 2: Demonstration semantic segmentation on 200 YAM episodes from 10 held-out tasks. Left: Boundary F1@

\pm

0.5/1/2 s and Grounded F1 using lexical or embedding-based matching. FAST, OAT, MotionGPT, and KiRoSe use the same LoRA-adapted Qwen3.5-9B reader; baseline input and training-set differences are listed in Appendix C . † Video and task instruction. Right: Boundary F1@

\pm

1 s as function of motion-token rate.

Figure 3: Evaluation Dataset. Top: example tasks and annotated action timelines, with colors indicating action categories. Bottom: representative 40-second traces of linear speed, angular speed, and opening rate for the left (black) and right (orange) arms or hands. The datasets differ in task semantics, temporal structure, and motion characteristics.

Figure 4: Boundary F1 (

\pm

0.5 s) per embodiment. KiRoSe is trained on YAM only; AgiBot and human hands are zero-shot.

Task	Traj.	Hits	Supplied words
Omelet	36	17	pan, egg, mixture, whisk
COVID-19 test	65	14	swab
Tomato & eggs	12	11	pan
Flat tire	13	8	inner (tube), lever
Noodles	9	3	pot, stir
Salad	26	2	toss, dressing

Table 3: Unseen-task vocabulary on Ego-Exo4D. Supplied words appear in the task’s annotations but not in YAM training or the prompt. Hits: trajectories with a supplied word.

Figure 5: Qualitative reversal probe on AgiBot (zero-shot). One subtask per episode is played backwards in place (red frames; stamps give original recording time). Curves show the signed progress rate (green: rising, red: falling); dashed: same model on the original video. ReWiND keeps counting up while the task is undone; KiRoSe’s progress falls; Table 4 reports statistic result.

Model	Fwd. $\rho\,\uparrow$	Rev. $\rho\,\downarrow$	Rev. $\rho<0$ (%) $\uparrow$
ReWiND (MW)	$\mathbf{+0.88}$	$+0.88$	1%
ReWiND (OXE)	$+0.65$	$+0.63$	4%
Robometer-4B	$+0.71$	$+0.06$	46%
KiRoSe	$+0.51$	$\mathbf{-0.68}$	86%

Table 4: Zero-shot reversal progress evaluation on AgiBot. The same 2,586 subtasks (357 episodes) are evaluated forward and reversed. “Rev.

\rho<0

” is the percentage of reversed subtasks with negative progress correlation.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training annotations (YAM, ABC-130K). Keyframes and the human subtask annotation of three training episodes, colored by leading verb as in Figures 7 – 9 . Annotations are dense and follow the arms’ kinematic events: alternating grasp and place cycles when organizing bottles, open, scoop, and close phases when filling the litter box, and explicit mistake intervals (hatched) when a grasp fails.

Figure 7: Segmentation examples on held-out YAM tasks (ABC-130K). The zero-shot video model tends to emit a few coarse segments that restate the instruction (“fold and stack blue towel”, “attempt insert plug”), whereas KiRoSe recovers the repeated grasp–place, flatten–fold, and insert cycles of the reference from kinematics alone. KiRoSe’s Boundary F1 at $\pm 1$ s exceeds the video baseline on all five episodes (0.44–0.72 versus 0.07–0.38).

Figure 8: Segmentation examples on AgiBot World (zero-shot embodiment transfer). Neither the motion tokenizer nor the reader sees AgiBot data, yet KiRoSe places boundaries at the grasp and release events and reaches Boundary F1 of 0.73–0.78 on all five episodes, above the video baseline on each (0.00–0.57). Its descriptions transfer at the level of actions (“pick up”, “place”), while object names often follow YAM phrasing.

Figure 9: Segmentation examples on Ego-Exo4D human-hand trajectories (zero-shot). Keyframes are from the wearer’s Aria camera (top 20% cropped). Overlays show the hand joints that the conversion of Appendix D uses, projected from the 3D annotations into the image with an approximate equidistant fisheye model: the wrist (dot), which supplies linear velocity; the wrist-to-knuckle fan, which defines the palm frame and angular velocity; and the dotted thumb–index line, whose length supplies the aperture. Frames without an overlay for a hand have no valid 3D track at that instant. Although KiRoSe never sees human data and hands have no gripper, it matches or exceeds the video baseline on all five episodes; over all 114 trajectories with video, KiRoSe reaches 0.375 Boundary F1 at $\pm 1$ s versus 0.255, and the gain depends on the motion tokens (Appendix D ).

Figure 10: Dilated temporal convolutions in the kinematic encoder. (a) Each residual block applies a kernel-5 convolution whose dilation doubles from block to block; tracing one output of the fourth block back to the input shows that the same five taps per layer cover 61 input samples. Padding is symmetric, so every feature sees both past and future motion. (b) With seven blocks (dilations

1,\ldots,64

), the receptive field grows to

1+4\sum_{i=0}^{6}2^{i}=509

samples, or 17 s at 30 Hz. This spans several average-length YAM segments, so each feature can encode where the current motion sits relative to neighboring subtasks, while the parameter count grows only linearly with depth.

Input	$\mathbf{x}_{t}\in\mathbb{R}^{14}$ at 30 Hz, per-episode z-scored
Projection	Conv1d( $14\!\rightarrow\!256$ , $k{=}1$ )
Residual blocks	7 $\times$ [Conv1d( $256$ , $k{=}5$ , dilation $2^{i}$ ) $\rightarrow$ GELU $\rightarrow$ Dropout(0.1) $\rightarrow$ Conv1d( $256$ , $k{=}1$ )]
Dilations	1, 2, 4, 8, 16, 32, 64 (symmetric padding)
Output	LayerNorm(256); receptive field 509 samples (17.0 s)
Encoder parameters	2,760,448
Boundary head grid	0.25 s; box pools of half-width 0.25, 1, 4 s, concatenated (768-d)

Appendix

Table 5: Kinematic encoder, transition supervision, and tokenizer settings.

Figure 11: Transition-supervision loss. (a) Soft targets on the 0.25 s candidate grid for boundaries at 2.0, 5.3, and 6.1 s (

\sigma=0.4

s). A grid point takes the height of the nearest Gaussian bump, so nearby boundaries merge instead of summing above one. (b) The focal weight

(1-p_{t})^{2}

against the model’s confidence in the target direction,

p_{t}

. The three marked points are the worked examples of Table 6 : an already-confident interior point is down-weighted by a factor of 400, while an under-predicted boundary keeps half its weight.

Grid point	$s$	$p$	BCE	$(1-p_{t})^{2}$	Contribution
Interior, already confident	0	0.05	0.051	0.0025	0.0001
Boundary, under-predicted	1	0.30	1.204	0.49	0.59
0.5 s from a boundary, over-predicted	0.46	0.90	1.292	0.283	0.366

Appendix

Table 6: Worked focal-loss examples. The gradient is carried by grid points near boundaries that the model is still unsure about.

Figure 12: How each robot channel is built for a robot and for a human hand. Rows are the channels of the shared representation (middle column). Dashed boxes: what each dataset provides; filled boxes: the conversion applied. Robots (left) need only forward kinematics, differentiation, and gripper normalization; human hands (right) require choosing a body frame, fitting a palm frame to the hand keypoints, using the thumb–index distance as a gripper, and projecting keystep annotations onto tracked intervals. The gap thresholds illustrated here describe the short-run conversion; the segmentation evaluation uses the longer-run protocol below.

Figure 13: Human hands as a robot end effector, on real data. (a) Four tasks, one single-frame pose each: the palm frame of each hand (red along the palm, green across, blue normal) and the thumb–index pinch (purple) that serves as the gripper, projected from 3D into the Aria image with an approximate equidistant fisheye model. Frames are shown only where all 14 joints the conversion uses agree with a manual 3D label within 2 cm. (b) The resulting channels for a COVID-test run next to a YAM episode of the same length; dotted lines are annotated boundaries. Wrist speed and aperture rate rise at keystep boundaries much as robot speed and gripper rate do at subtask boundaries. (c) Annotation projection: keystep starts that fall inside a tracked run become that run’s boundaries, in run time.

	YAM (ABC-130K)	AgiBot G1	Human hands (Ego-Exo4D)
Pose source	Forward kinematics from 6 joint angles (MJCF chain of the YAM model)	Recorded end-effector position and orientation ( state/end )	3D hand keypoints (wrist, MCPs, thumb/index tips)
Frame	Per-arm base frame (z up, x forward)	Robot base frame (z up)	World frame, rotated by the smoothed head yaw
$\mathbf{v}$	Central differences of position	Central differences of position	Central differences of 2 Hz low-passed wrist position
$\boldsymbol{\omega}$	From quaternion derivative	From quaternion derivative	From quaternion derivative of fitted palm frame
$g$	Normalized gripper aperture (0 closed, 1 open)	Effector position, affinely mapped to $[0,1]$ and inverted	Thumb-tip–index-tip distance, normalized per trajectory
Native rate	Several hundred Hz	$\approx$ 30 Hz	30 Hz (dense pseudo-ground truth)

Appendix

Table 8: Construction of the shared 14-D representation per data source.

Model	AgiBot test tasks	AgiBot held-out task	YAM held-out tasks
ReWiND (MW)	$+0.85$ / $+0.85$	$+0.89$ / $+0.89$	$+0.79$ / $+0.79$
ReWiND (OXE)	$+0.67$ / $+0.69$	$+0.65$ / $+0.62$	$+0.77$ / $+0.78$
Robometer-4B	$+0.68$ / $+0.08$	$+0.72$ / $+0.05$	$+0.53$ / $+0.30$
KiRoSe	$+0.50$ / $\mathbf{-0.75}$	$+0.51$ / $\mathbf{-0.66}$	$+0.64$ / $\mathbf{-0.66}$

Appendix

Table 9: Progress correlation by evaluation split , forward / reversed mean Spearman

\rho

. KiRoSe is the signed-rate head. Pooled over all 2,586 AgiBot subtasks, KiRoSe obtains

+0.51\,[0.48,0.53]

forward and

-0.68\,[-0.70,-0.67]

reversed (95% episode-clustered bootstrap intervals), with

\rho<0

on 86% of reversed subtasks.

Base model	Qwen3.5-9B, bfloat16, gradient checkpointing
LoRA	rank 16, $\alpha=32$ , dropout 0.05; q, k, v, o, gate, up, down projections
Trainable	LoRA adapters + 512 new input-embedding and output rows
Optimizer	AdamW, lr $2\!\times\!10^{-4}$ (constant, no warm-up), weight decay 0.01, grad-norm clip 1.0
Batch	1 episode per GPU, global batch 9 (3 GPUs, data parallel)
Data / epochs	5,478 YAM training episodes / 1 epoch
Compute	$\approx$ 2.2 h on 3 NVIDIA H200 GPUs

Appendix

Table 10: Reader training and inference settings.

Motion Segmentation

See all 17 papers

Mar 2, 2026

ROSER: Few-Shot Robotic Sequence Retrieval for Scalable Robot Learning

A critical bottleneck in robot learning is the scarcity of task-labeled, segmented training data, despite the abundance of large-scale robotic datasets recorded as long, continuous interaction logs. Existing datasets contain vast amounts of diverse behaviors, yet remain structurally incompatible with modern learning frameworks that require cleanly segmented, task-specific trajectories. We address this data utilization crisis by formalizing robotic sequence retrieval: the task of extracting reusable, task-centric segments from unlabeled logs using only a few reference examples. We introduce ROSER, a lightweight few-shot retrieval framework that learns task-agnostic metric spaces over temporal windows, enabling accurate retrieval with as few as 3-5 demonstrations, without any task-specific training required. To validate our approach, we establish comprehensive evaluation protocols and benchmark ROSER against classical alignment methods, learned embeddings, and language model baselines across three large-scale datasets (e.g., LIBERO, DROID, and nuScenes). Our experiments demonstrate that ROSER consistently outperforms all prior methods in both accuracy and efficiency, achieving sub-millisecond per-match inference while maintaining superior distributional alignment. By reframing data curation as few-shot retrieval, ROSER provides a practical pathway to unlock underutilized robotic datasets, fundamentally improving data availability for robot learning.

Zillur Rahman, Eddison Pham, Alejandro Daniel Noel +1

University of Nevada, Las Vegas · University of Toronto · LatentWorlds AI, NL

Data-Efficient Robot Learning Few-Shot Learning Metric Learning

Ask PDF

Apr 8, 2026

KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis

We present KITE, a training-free, keyframe-anchored, layout-grounded front-end that converts long robot-execution videos into compact, interpretable tokenized evidence for vision-language models (VLMs). KITE distills each trajectory into a small set of motion-salient keyframes with open-vocabulary detections and pairs each keyframe with a schematic bird's-eye-view (BEV) representation that encodes relative object layout, axes, timestamps, and detection confidence. These visual cues are serialized with robot-profile and scene-context tokens into a unified prompt, allowing the same front-end to support failure detection, identification, localization, explanation, and correction with an off-the-shelf VLM. On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL in the training-free setting, with especially large gains on simulation failure detection, identification, and localization, while remaining competitive with a RoboFAC-tuned baseline. A small QLoRA fine-tune further improves explanation and correction quality. We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE as a structured and interpretable front-end for robot failure analysis. Code and models are released on our project page: https://m80hz.github.io/kite/

Mehdi Hosseinzadeh, King Hang Wong, Feras Dayoub

Australian Institute for Machine Learning (AIML), Adelaide University, Australia

Robot Failure Detection VLMs for Robotics

Ask PDF

Aug 25, 2026

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling

Discrete motion representations have substantially advanced autoregressive text-to-motion generation. However, most motion tokenizers are optimized for reconstruction and do not explicitly allocate capacity according to semantic role. Action-level meaning and fine-grained kinematic detail must therefore be encoded through the same reconstruction-driven hierarchy. We introduce SeMoCo, a semantic-first motion codec, together with a dual-axis motion generator for language-conditioned motion generation. Each motion token contains one semantic token and a residual sequence of kinematic tokens. The generator models semantic progression across time and autoregressively refines the residual entries. We also construct

Ω

-MotionVerse, a large-scale, multi-source human-motion dataset unified under the SOMA representation. Across the reported comparisons, SeMoCo achieves the best reconstruction accuracy among the compared codecs, while strong text-to-motion results demonstrate the effectiveness of its motion tokens for downstream generation.

Tianlv Huang, Hetian Guo, Ziyi Cai +6

Jilin University · Frontier Robotics · Harbin Institute of Technology, Shenzhen

Text-to-Motion Generation Human Motion Generation Motion Representation Learning

Ask PDF

Abstract

Motion Segmentation

Explore similar work

ROSER: Few-Shot Robotic Sequence Retrieval for Scalable Robot Learning

KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling