cs.ROOct 22, 2026
SaveKiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations
Organizations: University of California, Berkeley · Toyota Research Institute
Abstract
Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: https://berkeleyautomation.github.io/kirose/)
Figures & tables
Figure 1: Encoding Kinematic Events into Sparse Motion Tokens. Across diverse manipulation tasks and objects, continuous robot motion decomposes into recurring, temporally localized patterns termed kinematic events (e.g., bimanual lift, handover, pick, place). Despite visual and spatial variations, end-effector trajectories within the same motion primitive share structural kinematic signatures. KiRoSe compresses kinematic events into a short, ordered sequence of discrete Sparse Motion Tokens (colored blocks; shaded chronologically).
Figure 2: KiRoSe Pipeline. Given a robot kinematic stream and the task instruction “fold the paper box,” KiRoSe encodes motion using a transition-supervised encoder, then temporally pools and quantizes its features into Sparse Motion Tokens with a learned codebook. The language model reads these tokens, time markers, and the instruction to output action descriptions with start and end times, such as “pick up the object,” “place in the box,” and “adjust.” Transition annotations provide training supervision, while the motion tokenizer remains frozen during language-model adaptation.
Table 2: Demonstration semantic segmentation on 200 YAM episodes from 10 held-out tasks. Left: Boundary F1@ 0.5/1/2 s and Grounded F1 using lexical or embedding-based matching. FAST, OAT, MotionGPT, and KiRoSe use the same LoRA-adapted Qwen3.5-9B reader; baseline input and training-set differences are listed in Appendix C . † Video and task instruction. Right: Boundary F1@ 1 s as function of motion-token rate.
Figure 3: Evaluation Dataset. Top: example tasks and annotated action timelines, with colors indicating action categories. Bottom: representative 40-second traces of linear speed, angular speed, and opening rate for the left (black) and right (orange) arms or hands. The datasets differ in task semantics, temporal structure, and motion characteristics.
Figure 4: Boundary F1 ( 0.5 s) per embodiment. KiRoSe is trained on YAM only; AgiBot and human hands are zero-shot.
| Task | Traj. | Hits | Supplied words |
| Omelet | 36 | 17 | pan, egg, mixture, whisk |
| COVID-19 test | 65 | 14 | swab |
| Tomato & eggs | 12 | 11 | pan |
| Flat tire | 13 | 8 | inner (tube), lever |
| Noodles | 9 | 3 | pot, stir |
| Salad | 26 | 2 | toss, dressing |
Table 3: Unseen-task vocabulary on Ego-Exo4D. Supplied words appear in the task’s annotations but not in YAM training or the prompt. Hits: trajectories with a supplied word.
Figure 5: Qualitative reversal probe on AgiBot (zero-shot). One subtask per episode is played backwards in place (red frames; stamps give original recording time). Curves show the signed progress rate (green: rising, red: falling); dashed: same model on the original video. ReWiND keeps counting up while the task is undone; KiRoSe’s progress falls; Table 4 reports statistic result.
| Model | Fwd. | Rev. | Rev. (%) |
| ReWiND (MW) | 1% | ||
| ReWiND (OXE) | 4% | ||
| Robometer-4B | 46% | ||
| KiRoSe | 86% |
Table 4: Zero-shot reversal progress evaluation on AgiBot. The same 2,586 subtasks (357 episodes) are evaluated forward and reversed. “Rev. ” is the percentage of reversed subtasks with negative progress correlation.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training annotations (YAM, ABC-130K). Keyframes and the human subtask annotation of three training episodes, colored by leading verb as in Figures 7 – 9 . Annotations are dense and follow the arms’ kinematic events: alternating grasp and place cycles when organizing bottles, open, scoop, and close phases when filling the litter box, and explicit mistake intervals (hatched) when a grasp fails.
Figure 7: Segmentation examples on held-out YAM tasks (ABC-130K). The zero-shot video model tends to emit a few coarse segments that restate the instruction (“fold and stack blue towel”, “attempt insert plug”), whereas KiRoSe recovers the repeated grasp–place, flatten–fold, and insert cycles of the reference from kinematics alone. KiRoSe’s Boundary F1 at s exceeds the video baseline on all five episodes (0.44–0.72 versus 0.07–0.38).
Figure 8: Segmentation examples on AgiBot World (zero-shot embodiment transfer). Neither the motion tokenizer nor the reader sees AgiBot data, yet KiRoSe places boundaries at the grasp and release events and reaches Boundary F1 of 0.73–0.78 on all five episodes, above the video baseline on each (0.00–0.57). Its descriptions transfer at the level of actions (“pick up”, “place”), while object names often follow YAM phrasing.
Figure 9: Segmentation examples on Ego-Exo4D human-hand trajectories (zero-shot). Keyframes are from the wearer’s Aria camera (top 20% cropped). Overlays show the hand joints that the conversion of Appendix D uses, projected from the 3D annotations into the image with an approximate equidistant fisheye model: the wrist (dot), which supplies linear velocity; the wrist-to-knuckle fan, which defines the palm frame and angular velocity; and the dotted thumb–index line, whose length supplies the aperture. Frames without an overlay for a hand have no valid 3D track at that instant. Although KiRoSe never sees human data and hands have no gripper, it matches or exceeds the video baseline on all five episodes; over all 114 trajectories with video, KiRoSe reaches 0.375 Boundary F1 at s versus 0.255, and the gain depends on the motion tokens (Appendix D ).
Figure 10: Dilated temporal convolutions in the kinematic encoder. (a) Each residual block applies a kernel-5 convolution whose dilation doubles from block to block; tracing one output of the fourth block back to the input shows that the same five taps per layer cover 61 input samples. Padding is symmetric, so every feature sees both past and future motion. (b) With seven blocks (dilations ), the receptive field grows to samples, or 17 s at 30 Hz. This spans several average-length YAM segments, so each feature can encode where the current motion sits relative to neighboring subtasks, while the parameter count grows only linearly with depth.
| Input | at 30 Hz, per-episode z-scored |
| Projection | Conv1d( , ) |
| Residual blocks | 7 [Conv1d( , , dilation ) GELU Dropout(0.1) Conv1d( , )] |
| Dilations | 1, 2, 4, 8, 16, 32, 64 (symmetric padding) |
| Output | LayerNorm(256); receptive field 509 samples (17.0 s) |
| Encoder parameters | 2,760,448 |
| Boundary head grid | 0.25 s; box pools of half-width 0.25, 1, 4 s, concatenated (768-d) |
Table 5: Kinematic encoder, transition supervision, and tokenizer settings.
Figure 11: Transition-supervision loss. (a) Soft targets on the 0.25 s candidate grid for boundaries at 2.0, 5.3, and 6.1 s ( s). A grid point takes the height of the nearest Gaussian bump, so nearby boundaries merge instead of summing above one. (b) The focal weight against the model’s confidence in the target direction, . The three marked points are the worked examples of Table 6 : an already-confident interior point is down-weighted by a factor of 400, while an under-predicted boundary keeps half its weight.
| Grid point | BCE | Contribution | |||
| Interior, already confident | 0 | 0.05 | 0.051 | 0.0025 | 0.0001 |
| Boundary, under-predicted | 1 | 0.30 | 1.204 | 0.49 | 0.59 |
| 0.5 s from a boundary, over-predicted | 0.46 | 0.90 | 1.292 | 0.283 | 0.366 |
Table 6: Worked focal-loss examples. The gradient is carried by grid points near boundaries that the model is still unsure about.
Figure 12: How each robot channel is built for a robot and for a human hand. Rows are the channels of the shared representation (middle column). Dashed boxes: what each dataset provides; filled boxes: the conversion applied. Robots (left) need only forward kinematics, differentiation, and gripper normalization; human hands (right) require choosing a body frame, fitting a palm frame to the hand keypoints, using the thumb–index distance as a gripper, and projecting keystep annotations onto tracked intervals. The gap thresholds illustrated here describe the short-run conversion; the segmentation evaluation uses the longer-run protocol below.
Figure 13: Human hands as a robot end effector, on real data. (a) Four tasks, one single-frame pose each: the palm frame of each hand (red along the palm, green across, blue normal) and the thumb–index pinch (purple) that serves as the gripper, projected from 3D into the Aria image with an approximate equidistant fisheye model. Frames are shown only where all 14 joints the conversion uses agree with a manual 3D label within 2 cm. (b) The resulting channels for a COVID-test run next to a YAM episode of the same length; dotted lines are annotated boundaries. Wrist speed and aperture rate rise at keystep boundaries much as robot speed and gripper rate do at subtask boundaries. (c) Annotation projection: keystep starts that fall inside a tracked run become that run’s boundaries, in run time.
| YAM (ABC-130K) | AgiBot G1 | Human hands (Ego-Exo4D) | |
| Pose source | Forward kinematics from 6 joint angles (MJCF chain of the YAM model) | Recorded end-effector position and orientation ( state/end ) | 3D hand keypoints (wrist, MCPs, thumb/index tips) |
| Frame | Per-arm base frame (z up, x forward) | Robot base frame (z up) | World frame, rotated by the smoothed head yaw |
| Central differences of position | Central differences of position | Central differences of 2 Hz low-passed wrist position | |
| From quaternion derivative | From quaternion derivative | From quaternion derivative of fitted palm frame | |
| Normalized gripper aperture (0 closed, 1 open) | Effector position, affinely mapped to and inverted | Thumb-tip–index-tip distance, normalized per trajectory | |
| Native rate | Several hundred Hz | 30 Hz | 30 Hz (dense pseudo-ground truth) |
Table 8: Construction of the shared 14-D representation per data source.
| Model | AgiBot test tasks | AgiBot held-out task | YAM held-out tasks |
| ReWiND (MW) | / | / | / |
| ReWiND (OXE) | / | / | / |
| Robometer-4B | / | / | / |
| KiRoSe | / | / | / |
Table 9: Progress correlation by evaluation split , forward / reversed mean Spearman . KiRoSe is the signed-rate head. Pooled over all 2,586 AgiBot subtasks, KiRoSe obtains forward and reversed (95% episode-clustered bootstrap intervals), with on 86% of reversed subtasks.
| Base model | Qwen3.5-9B, bfloat16, gradient checkpointing |
| LoRA | rank 16, , dropout 0.05; q, k, v, o, gate, up, down projections |
| Trainable | LoRA adapters + 512 new input-embedding and output rows |
| Optimizer | AdamW, lr (constant, no warm-up), weight decay 0.01, grad-norm clip 1.0 |
| Batch | 1 episode per GPU, global batch 9 (3 GPUs, data parallel) |
| Data / epochs | 5,478 YAM training episodes / 1 epoch |
| Compute | 2.2 h on 3 NVIDIA H200 GPUs |
Table 10: Reader training and inference settings.