Temporal Action Segmentation

Follow this topic Ask about this topic

Temporal action segmentation assigns action labels and boundaries to frames in long, untrimmed videos, making it possible to analyze how procedures and everyday activities unfold. Methods include temporal convolutional networks and transformers, hierarchical models, sequence decoding, and optimal-transport approaches; research also explores weakly supervised, unsupervised, skeleton-based, and vision-language methods. Systems are evaluated on benchmarks such as Breakfast, 50Salads, GTEA, and Assembly101, as well as surgical, egocentric, and robotic workflows. A central challenge is locating precise boundaries without excessive labels while avoiding fragmented predictions and maintaining performance across settings, viewpoints, and online use.

Latest papers

49 papers1 paper in the last four weeks, down 83% on the four weeks before. 0.0% of all new papers.

KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations
Oct 22, 2026Shuangyu Xie, Kaiyuan Chen, Muhammad Zubair Irshad +3Motion Segmentation Temporal Action Segmentation
University of California, Berkeley · Toyota Research Institute
Groupwise Selective State-Space Filtering for Accurate and Streaming Action Boundary Detection
Sep 27, 2026Mustafa Bora ÇelikTemporal Action Segmentation Selective SSMs
Department of Electrical and Electronics Engineering, Ankara Medipol University, Ankara, Turkey
Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos
Sep 25, 2026Bowen Guo, Shiwei Gan, Yafeng Yin +4Boundary-Aware Segmentation Temporal Action Segmentation
State Key Laboratory for Novel Software Technology, Nanjing University, Suzhou, China
CE $^4$ L: Continual Ego, Exo, and Ego-Exo Learning
Sep 20, 2026Hongwei Yan, Kanglei Zhou, Yuchen Liu +3Temporal Action Segmentation Next-Action Prediction
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Tsinghua-Peking Center for Life Sciences · Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China +2
Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation
Sep 8, 2026Runzhong Zhang, Yueqi Duan, Yang Chen +4Temporal Action Segmentation Boundary-Aware Segmentation
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Electronic Engineering, Tsinghua University, Beijing, China · Amazon, Seattle, America
An Integrated Video-AI Platform for Action-Level Microanastomosis Training and Performance Feedback
Sep 6, 2026Yan Meng, Daniel A. DonohoSurgical Video Understanding Temporal Action Segmentation
Department of Neurosurgery, Children’s National Hospital
A Framework for Egocentric and Exocentric Procedural Understanding via Temporal Segmentation and Semantic Abstraction
Sep 4, 2026Vivek Chavan, Jörg KrügerMultimodal Memory Egocentric Video Understanding
Fraunhofer IPK, Berlin, Germany · Technische Universität Berlin, Berlin, Germany
FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentation
Aug 30, 2026Linxiang Peng, Xinyao Qin, Jinhan Li +2Temporal Action Segmentation Optimal Transport
Suzhou Institute for Advanced Research, University of Science and Technology of China · School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Big Data & AI Research and Engineering Center +1
MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation
Aug 30, 2026Xinyao Qin, Linxiang Peng, Youbao Ye +2Temporal Action Segmentation Unsupervised Representation Learning
University of Science and Technology of China, China
See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning
Aug 30, 2026Yun Li, Jun Xiao, Cong Zhang +1Temporal Action Segmentation Temporal Representation Learning
School of Control Science and Engineering, Shandong University
VidParse: Online Parsing of Egocentric Procedures Like a Pro
Aug 27, 2026Anubhav Gupta, Archit Kambhamettu, Vatsal Agarwal +2Video Understanding Egocentric Video Understanding
University of Maryland, College Park, USA
ConsensusTAS: Self-Supervised Temporal Action Segmentation for Long-Horizon Construction Videos
Aug 25, 2026Xiaoshan Zhou, Yafei SunTemporal Action Segmentation Human Activity Recognition
School of Project Management, University of Sydney, Australia · School of Civil and Environmental Engineering, University of New South Wales, Australia
Artificial Intelligence for Workflow Analysis in Colorectal Surgery: A Multicentric, Cross-Procedural Development and Generalization Study
Aug 20, 2026Pietro Mascagni, Julia Alekseenko, Pooja P Jain +10Surgical Video Understanding Temporal Action Segmentation
IHU Strasbourg, Strasbourg, France · Surgical Video Analysis, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy · Università Cattolica del Sacro Cuore, Rome, Italy +4
D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation
Aug 6, 2026Elena Bueno-Benito, Mariella DimiccoliTemporal Action Segmentation Optimal Transport
Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC, Barcelona, Spain
Lag-aware cross-hand alignment for dual-hand action segmentation
Jul 28, 2026Fatemeh ZiaeetabarTemporal Action Segmentation Boundary-Aware Segmentation
Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran
Adaptive Latent Trajectory Anchoring for Action Segmentation Dataset Condensation
Jul 10, 2026Artheme Gauthier-Villar, Guodong Ding, Angela YaoTemporal Action Segmentation Dataset Condensation
ETH Zurich, Switzerland · National University of Singapore, Singapore
Learning Probabilistic Embeddings for Unsupervised Action Segmentation
Jul 6, 2026Shuai Li, Duc Manh Vu, Juergen GallTemporal Action Segmentation Representation Learning
University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Grammar-Guided Hierarchical Parsing for Long-form Audio Activity Recognition
Jun 26, 2026Peng Zhang, Qingyu Luo, Philip J. B. Jackson +1Temporal Action Segmentation Human Activity Recognition
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, U.K.
P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture
Jun 22, 2026Felix Tristram, Stefano Gasperini, Benjamin Killeen +4Temporal Action Segmentation Video Understanding
Technical University of Munich Munich Center for Machine Learning · Carl Zeiss AG · Carl Zeiss AG[cs.CV]
The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection
Jun 19, 2026Serdar Ozsoy, Lars Doorenbos, Federico Spurio +2Temporal Action Segmentation Vision-Language Models
University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe
OR-Action: Multi-Role Video Understanding with Fine-Grained Actions
Jun 11, 2026Felix Tristram, Ege Özsoy, Christian Benz +3Temporal Action Segmentation Human Activity Recognition
Technical University of Munich, Germany · Munich Center for Machine Learning (MCML) · Carl Zeiss AG, Germany
Intention Driven Identification of In-Possession Match Phases in Association Football through Temporal Graph Learning
Jun 8, 2026Yuesen Li, Daniel LinkTemporal Action Segmentation Temporal GNNs
Technical University of Munich, Experimental Exercise Science, Munich, Germany · Technical University of Munich, Munich Data Science Institute, Munich, Germany
Uncertainty-Aware Action Segmentation for Human-to-Robot Assembly Teleoperation
Jun 6, 2026Fnu Heman, Yixuan Wang, Kolin Xu +7Temporal Action Segmentation Robot Teleoperation
Mechanical Engineering, University of California, Riverside, CA, USA. · Electrical and Computer Engineering, University of Miami, FL, USA.
VOLT: Vision and Language Trajectory Segmentation for Faster-than-Demonstration Policies
Jun 4, 2026Robert Ramirez Sanchez, Daniel J. Evans, Dylan P. Losey +1Temporal Action Segmentation Imitation Learning
Collab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA · Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA
Ego-METAS: Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark
May 29, 2026Maria Santos-Villafranca, Jesus Bermudez-cameo, Alejandro Perez-Yus +2Temporal Action Segmentation Energy-Efficient ML
University of Zaragoza - I3A · Department of Mathematics and Computer Science, University of Catania
Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioning
May 29, 2026Hao Zheng, Hu Wang, Tiantian Zheng +2Temporal Action Segmentation Vision Transformer
New York University Abu Dhabi, Abu Dhabi, United Arab Emirates · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Visualizing Latent Phase Structures in Locomotion Policies: A Multi-Environment Study with Temporal Feature Extension
May 27, 2026Daisuke Yasui, Toshitaka Matuki, Hiroshi SatoTemporal Action Segmentation Robot Locomotion
Mathematics and Computer Science National Defense Academy of Japan Yokosuka, Japan
EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation
May 18, 2026Rosario Leonardi, Francesco Ragusa, Daniele Materia +4Temporal Action Segmentation Next-Action Prediction
Department of Mathematics and Computer Science, University of Catania, Italy · Next Vision s.r.l., Catania, Italy · Reality Labs Research, Meta, USA
Improving Temporal Action Segmentation via Constraint-Aware Decoding
May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action Segmentation Constrained Decoding
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1
IMPACT-Scribe: Interactive Temporal Action Segmentation with Boundary Scribbles and Query Planning
May 3, 2026Qian Yin, Di Wen, Kunyu Peng +11Temporal Action Segmentation Human-in-the-Loop Annotation
Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany · INSAIT, Sofia University “St. Kliment Ohridski”, 1784 Sofia, Bulgaria · ETH Zurich, 8092 Zurich, Switzerland +1

Oct 22, 2026J/K move · Enter open · S save

KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

Shuangyu Xie, Kaiyuan Chen, Muhammad Zubair Irshad, Richard Cheng+2

University of California, Berkeley · Toyota Research Institute

Motion Segmentation Temporal Action Segmentation Motion Representation Learning

Open paper AskPDF arXiv

Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: https://berkeleyautomation.github.io/kirose/)