This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks, and adapt the Pix2Seq framework to learn autoregressive models to output these tokens. We propose novel tokenization strategies to compress the lengths of the token sequences to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our models on two domain-specific datasets to demonstrate their competitiveness with the state of the art in certain scenarios, in spite of being severely bottlenecked by our limited computational resources. We supplement these analyses by proposing several promising approaches to foster future competitiveness in general-purpose applications, and facilitate this by making our code and models publicly available.
Figures & tables
Figure 1: Generating RLE sequence for a 2×5 binary mask using row-major flattening. Note that the start indices use 0-based indexing instead of the 1-based indexing commonly used in RLE.
Figure 2: Visualization of RLE tokenization of binary segmentation masks with row-major (top) and column-major (bottom) flattening of the masks. Each figure shows (from left to right) the source image patch with the foreground mask drawn on it in yellow, binary version of this mask with the already tokenized segment in yellow, and the corresponding tokens. The run that is currently being tokenized is shown in purple. Animated versions of these figures are available here and here . Best viewed under high magnification.
Figure 3: An example of the sliding window patch extraction and mask subsampling process on an image from the IPSC dataset. The top row shows (from left right) source image resized to 2560×2560 with patch location shown by the blue box, corresponding mask at its full resolution of 2560×2560 , and this mask subsampled by a factor of 8 to 320×320 . The bottom row shows (from left right) 640×640 patch corresponding to the blue box, corresponding patch mask at its full resolution 640×640 , and this mask subsampled by a factor of 8 to 80×80 . This subsampled 80×80 mask is the one that is used for generating the RLE sequence. An animated version of this figure is available here .
Figure 4: Visualization of LAC tokenization for multi-class segmentation. The figure shows (from left to right) the source image patch with the two classes in red and green, binary version of this mask with the already tokenized segment in red or green depending on the class, and the corresponding tokens. The run that is currently being tokenized is shown in purple. The LAC tokens are shown here as concatenations of class name and length but each such combination represents a single unique token. Animated version of this figure is available here . Best viewed under high magnification.
Figure 5: Visualization of TAC tokenization for multi-class video segmentation with N=2 . The top row shows (from left to right) F1 , F2 , full resolution TAC mask, and subsampled TAC mask. The TAC masks show 8 TAC classes whose colors are shown at the top. The bottom row shows subsampled F1 and F2 masks, partially colored with TAC colors for runs whose tokens are shown on the right. Tokens are colored according to the TAC class in each run except the current one that is shown in purple. Animated version of this figure is available here . Best viewed under high magnification.
Table 1: TAC tokens for IPSC dataset with binary and multi-class masks for N=2 and N=3 . Here, bkg refers to the background while ips and dif are the two classes in the IPSC dataset. Both classes are represented by cell in the binary case. Note that there is no TAC token corresponding to the class combinations with bkg in every frame (i.e., bkg - bkg for N=2 and bkg - bkg - bkg for N=3 ) since RLE only represents foreground pixels, which requires that atleast one class be non- bkg . Also, N=3 multi-class case would have (C+1)N−1=33−1=26 TAC tokens, out of which only 15 are shown for brevity.
Table 2: Segmentation recall, precision and ice concentration median MAE on ARIS dataset for P2S-SEG and conventional deep learning models along with SVM. The fw in ice+water (fw) stands for frequency weighted. Relative increase over SVM in recall and precision is computed as (model_value−svm_value)/svm_value×100 while the relative decrease in median MAE is computed as (svm_mae−model_mae)/svm_mae×100 . The best and the second best models in each case are shown in bold and highlighted in green and yellow respectively. This data is shown as a bar plot in Fig. 17 .
Figure 6: Results of ablation tests with training images on ARIS dataset for (a-b) anchor ice and (c-d) frazil ice. Note the variable Y-axis limits between the top and bottom plots.
Figure 7: Semantic segmentation results on both (top) early and (bottom) late-stage IPSC datasets in terms of (left to right) recall, precision and dice score. IPSC and DfC on the x-axis refers to the two classes of cells while Cell represents the class-agnostic case where all the cells are considered as belonging to the same class.
Figure 8: Comparing the segmentation performance of models trained with and without frozen backbones on the (top) ARIS and (bottom) IPSC early-stage datasets. Models trained with and without frozen backbones are respectively shown in shades of green and red and denoted in the legend with prefixes F and T . The P2S-SEG suffixes patch and full respectively refer to models trained with and without sliding window patches. In the former case, P=640 , S=80 for both datasets, I=1280 for ARIS and I=2560 for IPSC. In the latter case, I=P=640 for both datasets, S=160 for ARIS and S=320 for IPSC.
Figure 9: Performance impact of replacing N video frames with only the first frame in the sequence as input to P2S-VIDSEG models. The two cases are denoted with V and S in the legend and shown in shades of red and green respectively. Results are shown for both (left) early and (right) late-stage IPSC datasets in terms of Dice score Recall and precision exhibited the same trends as Dice score and have therefore be relegated to Fig. 18 . Best viewed under high magnification.
Figure 10: Impact of video length N and stride T on P2S-VIDSEG performance over (left) early and (right) late-stage configurations of the IPSC dataset. T=1 and T=N are respectively represented with shades of red and green. Lighter shades of each colour represent N=2 while the darker shades represent N=8 . Dice score showed similar patterns as precision and has thus been relegated to Fig. 18 . Best viewed under high magnification.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: Statistics of static mask RLE lengths over the IPSC dataset with images resized from I=1280 to I=80 , without patches and mask subsampling (so that S=P=I ).
Table 4: Statistics of static mask RLE lengths over the entire IPSC dataset with (top left) I=P=640 , (top right) I=P=1024 , (bottom left) I=2560,P=640 , and (bottom right) I=2560,P=1024 .
Table 5: Statistics of video mask RLE lengths over the IPSC dataset with N=2 , I=2560 , P=640 and S=80,160 .
Table 6: Statistics of video mask RLE lengths over the IPSC dataset with I=2560 , P=640 , S=80 and N=3,4,6,8,9 .
Table 7: Vocabulary sizes ( V ) for video segmentation with S=40 , varying values of N from N=2 to N=20 and both binary and multiclass cases. The maximum possible N for each case such that V<32K is highlighted in yellow. V is shown in units of thousands.
Table 8: Vocabulary sizes ( V ) for video segmentation with S=80 , varying values of N from N=2 to N=16 and both binary and multiclass cases. The maximum possible N for each case such that V<32K is highlighted in yellow. V is shown in units of thousands.
Table 9: Vocabulary sizes ( V ) for video segmentation with S=128 , varying values of N from N=2 to N=16 and both binary and multiclass cases. The maximum possible N for each case such that V<32K is highlighted in yellow. V is shown in units of thousands.
Table 10: Vocabulary sizes ( V ) for video segmentation with S=160 , varying values of N from N=2 to N=16 and both binary and multiclass cases. The maximum possible N for each case such that V<32K is highlighted in yellow. V is shown in units of thousands.
Figure 11: Visualization of 3D-C RLE tokenization of binary video segmentation masks with N=2 , showing runs corresponding to the same part of the same object in F1 (left) and F2 (right). We can see that their tokens are completely unrelated in the sequence. Animated version of this figure is available here . Best viewed under high magnification.
Figure 12: Visualization of 3D-F RLE tokenization of binary video segmentation masks with N=2 . The left half shows the same part of the same object in F1 and F2 represented by the same run and the right half shows one of the unit sized runs created by a tiny change in the shape of the cell between the two frames. Animated version of this figure is available here . Best viewed under high magnification.
Table 11: Segmentation metrics representing mask quality degradation by subsampling, averaged over the entire ARIS and IPSC datasets.
Figure 13: Visualization of TAC tokenization for binary segmentation with N=2 . Animated version of this figure is available here . Best viewed under high magnification.
Figure 14: Visualization of TAC tokenization with N=3 for (top) binary and (bottom) multi-class masks. Animated versions of these figures are available here and here respectively. Best viewed under high magnification.
Figure 15: Visualization of LTAC tokenization for video segmentation masks with N=2 for (top) binary and (bottom) multi-class cases. Animated versions of these figures are available here and here respectively. Best viewed under high magnification.
Figure 16: Visualization of IW tokenization for multi-class static segmentation mask. The first image in the top row shows the class mask where IPSC and DfC cells are shown in green and red respectively. The second image shows the instance mask where each individual cell is shown in a different color. The last image shows the cell whose binary RLE tokens are currently being generated. The corresponding subsampled binary mask is shown in the first image in the second row. The tokens for the other two cells have already been generated (shown in corresponding colors) and terminated by the respective class tokens. An animated version of this figure is available here . Best viewed under high magnification.
Table 12: Configuration and training details for models whose results are reported in Sec. 5.3
Figure 17: Bar plot version of the data in Table 2 . Best viewed under high magnification.
Figure 18: Impact of (left and center) replacing N video frames with the first frame and (right) video length N and stride T on P2S-VIDSEG performance over (top) early and (bottom) late-stage configurations of the IPSC dataset.
Table 13: Segmentation metrics ( % ) on validation set while training P2S-SEG on IPSC early-stage dataset. Best viewed under high magnification.
Table 14: Segmentation metrics ( % ) on validation set while training P2S-VIDSEG on IPSC late-stage dataset. Best viewed under high magnification.
Table 15: Results of our best performing Cityscapes model on the validation set in terms of image-level, class-level and official metrics. Table 16 provides class and category-wise breakdown of the latter. This model was trained with the source images resized from 2048×1024 to 1280×640 , followed by sliding window patches of size P=640 . We used offline RLE with no data augmentation, LAC encoding, frozen backbone, S=80 , L=2K , V=8K , and B=80 .
Table 16: Class and category-wise breakdown of the results of our best performing Cityscapes model on the official cityscapes metrics. Please refer Table 15 for model details. Best viewed under high magnification.
Figure 19: An example of a multi-headed decoder architecture to output each RLE component by a separate head, while sharing the memory-expensive MHA module between them. Please refer to the baseline decoder in [ 17 , Fig. 3] that this adapts. This uses L=500 , S=80 , C=2 and naïve RLE encoding with 2D starts and separate length and class tokens. There are thus 4 tokens per run, represented by SX, SY, LEN and CLS, and shown respectively in blue, magenta, green and cyan. Compared to 1D starts and LAC with a single-headed decoder, this not only cuts L by half, but also reduces V by up to 3 orders of magnitude, since each component now has its own vocabulary which can be as small as S for coordinate tokens and C for class tokens.
Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4× its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.
Mikhail Dereviannykh, Vikram Voleti, Simon Donne +3
Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mismatch between the representation used during training and the compact interface required at deployment. We present CoVisco, a codec-native vision encoder with native token compression for unified image-video understanding. By combining codec-native input support with segmented attention, CoVisco can encode long visual inputs in a single forward pass without forming dense patch-to-patch interactions across all frames. Each temporal segment is equipped with learnable abstract tokens that learn a compact segment-level representation, while fine-grained patch tokens remain available throughout the encoder. Alternating intra-segment and abstract-communication layers preserve video-level context through the abstract-token channel. A lightweight selector further exposes either abstract tokens alone or abstract tokens augmented with a runtime-selected subset of patch tokens, yielding a compact visual interface that reduces the visual context and prefill burden of downstream MLLMs while retaining fine-grained evidence when needed. Pretrained with contrastive objectives on 565M image--text pairs and 6.4M videos, CoVisco shows competitive performance on video-oriented embedding and multimodal understanding benchmarks. In the evaluated four-segment, 64-frame setting, abstract-only inference uses only 400 visual tokens while achieving video-understanding performance close to, and on some benchmarks exceeding, OneVision-Encoder. Selected patch tokens further improve fine-grained video reasoning. Project URL: https://github.com/ernie-research/CoVisco.git
Yulong Liu, Xiaotian Han, Junyuan Shang +6
ERNIE Team, Baidu Inc. · The Hong Kong University of Science and Technology · Institute of Automation, Chinese Academy of Sciences (CASIA)
Recent advances in video object segmentation with Multimodal Large Language Model (MLLM) reasoning have demonstrated the effectiveness of using a single textual token, such as SEG, to predict segmentation masks across images and videos. However, we observe that this single-token strategy lacks the granularity required to precisely localize multiple objects across time in video segmentation tasks. To address this limitation, we develop Multi-Token Reasoning for Video Object Segmentation, or MoVISA. MoVISA uses multiple segmentation tokens, such as SEG0 and SEG1, to represent an object across different frames. This design enables more fine-grained alignment between language prompts and spatio-temporal mask predictions, improving both performance and interpretability. On the challenging MeViS, DAVIS17, ReVOS, and Ref-Youtube-VOS benchmarks, our model achieves a 13.2 percent J and F improvement on MeViS and an 8.4 percent J and F improvement on ReVOS. Code and models will be released.