Tokenizing Semantic Segmentation with Run Length Encoding

Abhineet Singh; Justin Rozeboom; Nilanjan Ray

cs.CVFeb 25, 2026

Tokenizing Semantic Segmentation with Run Length Encoding

Authors: Abhineet Singh, Justin Rozeboom, Nilanjan Ray

Organizations: Department of Computing Science, University of Alberta, Edmonton, Canada

Abstract

This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks, and adapt the Pix2Seq framework to learn autoregressive models to output these tokens. We propose novel tokenization strategies to compress the lengths of the token sequences to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our models on two domain-specific datasets to demonstrate their competitiveness with the state of the art in certain scenarios, in spite of being severely bottlenecked by our limited computational resources. We supplement these analyses by proposing several promising approaches to foster future competitiveness in general-purpose applications, and facilitate this by making our code and models publicly available.

Figures & tables

Figure 1: Generating RLE sequence for a $2\times 5$ binary mask using row-major flattening. Note that the start indices use 0-based indexing instead of the 1-based indexing commonly used in RLE.

Panel of Figure 2 — Figure 2: Visualization of RLE tokenization of binary segmentation masks with row-major (top) and column-major (bottom) flattening of the masks. Each figure shows (from left to right) the source image patch with the foreground mask drawn on it in yellow, binary version of this mask with the already tokenized segment in yellow, and the corresponding tokens. The run that is currently being tokenized is shown in purple. Animated versions of these figures are available here and here . Best viewed under high magnification.

Figure 3: An example of the sliding window patch extraction and mask subsampling process on an image from the IPSC dataset. The top row shows (from left right) source image resized to $2560\times 2560$ with patch location shown by the blue box, corresponding mask at its full resolution of $2560\times 2560$ , and this mask subsampled by a factor of 8 to $320\times 320$ . The bottom row shows (from left right) $640\times 640$ patch corresponding to the blue box, corresponding patch mask at its full resolution $640\times 640$ , and this mask subsampled by a factor of 8 to $80\times 80$ . This subsampled $80\times 80$ mask is the one that is used for generating the RLE sequence. An animated version of this figure is available here .

Figure 4: Visualization of LAC tokenization for multi-class segmentation. The figure shows (from left to right) the source image patch with the two classes in red and green, binary version of this mask with the already tokenized segment in red or green depending on the class, and the corresponding tokens. The run that is currently being tokenized is shown in purple. The LAC tokens are shown here as concatenations of class name and length but each such combination represents a single unique token. Animated version of this figure is available here . Best viewed under high magnification.

Figure 5: Visualization of TAC tokenization for multi-class video segmentation with $N=2$ . The top row shows (from left to right) $F_{1}$ , $F_{2}$ , full resolution TAC mask, and subsampled TAC mask. The TAC masks show 8 TAC classes whose colors are shown at the top. The bottom row shows subsampled $F_{1}$ and $F_{2}$ masks, partially colored with TAC colors for runs whose tokens are shown on the right. Tokens are colored according to the TAC class in each run except the current one that is shown in purple. Animated version of this figure is available here . Best viewed under high magnification.

Table 1: TAC tokens for IPSC dataset with binary and multi-class masks for

N=2

and

N=3

. Here, bkg refers to the background while ips and dif are the two classes in the IPSC dataset. Both classes are represented by cell in the binary case. Note that there is no TAC token corresponding to the class combinations with bkg in every frame (i.e., bkg - bkg for

N=2

and bkg - bkg - bkg for

N=3

) since RLE only represents foreground pixels, which requires that atleast one class be non- bkg . Also,

N=3

multi-class case would have

(C+1)^{N}-1=3^{3}-1=26

TAC tokens, out of which only

15

are shown for brevity.

Table 2: Segmentation recall, precision and ice concentration median MAE on ARIS dataset for P2S-SEG and conventional deep learning models along with SVM. The fw in ice+water (fw) stands for frequency weighted. Relative increase over SVM in recall and precision is computed as

(\texttt{model\_value}-\texttt{svm\_value})/\texttt{svm\_value}\times 100

while the relative decrease in median MAE is computed as

(\texttt{svm\_mae}-\texttt{model\_mae})/\texttt{svm\_mae}\times 100

. The best and the second best models in each case are shown in bold and highlighted in green and yellow respectively. This data is shown as a bar plot in Fig. 17 .

Figure 6: Results of ablation tests with training images on ARIS dataset for (a-b) anchor ice and (c-d) frazil ice. Note the variable Y-axis limits between the top and bottom plots.

Panel of Figure 7 — Figure 7: Semantic segmentation results on both (top) early and (bottom) late-stage IPSC datasets in terms of (left to right) recall, precision and dice score. IPSC and DfC on the x-axis refers to the two classes of cells while Cell represents the class-agnostic case where all the cells are considered as belonging to the same class.

Figure 8: Comparing the segmentation performance of models trained with and without frozen backbones on the (top) ARIS and (bottom) IPSC early-stage datasets. Models trained with and without frozen backbones are respectively shown in shades of green and red and denoted in the legend with prefixes F and T . The P2S-SEG suffixes patch and full respectively refer to models trained with and without sliding window patches. In the former case,

P=640

S=80

for both datasets,

I=1280

for ARIS and

I=2560

for IPSC. In the latter case,

I=P=640

for both datasets,

S=160

for ARIS and

S=320

for IPSC.

Figure 9: Performance impact of replacing

N

video frames with only the first frame in the sequence as input to P2S-VIDSEG models. The two cases are denoted with V and S in the legend and shown in shades of red and green respectively. Results are shown for both (left) early and (right) late-stage IPSC datasets in terms of Dice score Recall and precision exhibited the same trends as Dice score and have therefore be relegated to Fig. 18 . Best viewed under high magnification.

Figure 10: Impact of video length

N

and stride

T

on P2S-VIDSEG performance over (left) early and (right) late-stage configurations of the IPSC dataset.

T=1

and

T=N

are respectively represented with shades of red and green. Lighter shades of each colour represent

N=2

while the darker shades represent

N=8

. Dice score showed similar patterns as precision and has thus been relegated to Fig. 18 . Best viewed under high magnification.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Table 3: Statistics of static mask RLE lengths over the IPSC dataset with images resized from

I=1280

I=80

, without patches and mask subsampling (so that

S=P=I

Table 4: Statistics of static mask RLE lengths over the entire IPSC dataset with (top left)

I=P=640

, (top right)

I=P=1024

, (bottom left)

I=2560,P=640

, and (bottom right)

I=2560,P=1024

Table 5: Statistics of video mask RLE lengths over the IPSC dataset with

N=2

I=2560

P=640

and

S=80,160

Table 6: Statistics of video mask RLE lengths over the IPSC dataset with

I=2560

P=640

S=80

and

N=3,4,6,8,9

Table 7: Vocabulary sizes (

V

) for video segmentation with

S=40

, varying values of

N

from

N=2

N=20

and both binary and multiclass cases. The maximum possible

N

for each case such that

V<32K

is highlighted in yellow.

V

is shown in units of thousands.

Table 8: Vocabulary sizes (

V

) for video segmentation with

S=80

, varying values of

N

from

N=2

N=16

and both binary and multiclass cases. The maximum possible

N

for each case such that

V<32K

is highlighted in yellow.

V

is shown in units of thousands.

Table 9: Vocabulary sizes (

V

) for video segmentation with

S=128

, varying values of

N

from

N=2

N=16

and both binary and multiclass cases. The maximum possible

N

for each case such that

V<32K

is highlighted in yellow.

V

is shown in units of thousands.

Table 10: Vocabulary sizes (

V

) for video segmentation with

S=160

, varying values of

N

from

N=2

N=16

and both binary and multiclass cases. The maximum possible

N

for each case such that

V<32K

is highlighted in yellow.

V

is shown in units of thousands.

Panel of Figure 11 — Figure 11: Visualization of 3D-C RLE tokenization of binary video segmentation masks with $N=2$ , showing runs corresponding to the same part of the same object in $F_{1}$ (left) and $F_{2}$ (right). We can see that their tokens are completely unrelated in the sequence. Animated version of this figure is available here . Best viewed under high magnification.

Panel of Figure 12 — Figure 12: Visualization of 3D-F RLE tokenization of binary video segmentation masks with $N=2$ . The left half shows the same part of the same object in $F_{1}$ and $F_{2}$ represented by the same run and the right half shows one of the unit sized runs created by a tiny change in the shape of the cell between the two frames. Animated version of this figure is available here . Best viewed under high magnification.

Table 11: Segmentation metrics representing mask quality degradation by subsampling, averaged over the entire ARIS and IPSC datasets.

Figure 13: Visualization of TAC tokenization for binary segmentation with $N=2$ . Animated version of this figure is available here . Best viewed under high magnification.

Panel of Figure 14 — Figure 14: Visualization of TAC tokenization with $N=3$ for (top) binary and (bottom) multi-class masks. Animated versions of these figures are available here and here respectively. Best viewed under high magnification.

Panel of Figure 15 — Figure 15: Visualization of LTAC tokenization for video segmentation masks with $N=2$ for (top) binary and (bottom) multi-class cases. Animated versions of these figures are available here and here respectively. Best viewed under high magnification.

Figure 16: Visualization of IW tokenization for multi-class static segmentation mask. The first image in the top row shows the class mask where IPSC and DfC cells are shown in green and red respectively. The second image shows the instance mask where each individual cell is shown in a different color. The last image shows the cell whose binary RLE tokens are currently being generated. The corresponding subsampled binary mask is shown in the first image in the second row. The tokens for the other two cells have already been generated (shown in corresponding colors) and terminated by the respective class tokens. An animated version of this figure is available here . Best viewed under high magnification.

Table 12: Configuration and training details for models whose results are reported in Sec. 5.3

Panel of Figure 17 — Figure 17: Bar plot version of the data in Table 2 . Best viewed under high magnification.

Panel of Figure 18 — Figure 18: Impact of (left and center) replacing $N$ video frames with the first frame and (right) video length $N$ and stride $T$ on P2S-VIDSEG performance over (top) early and (bottom) late-stage configurations of the IPSC dataset.

Table 13: Segmentation metrics (

\%

) on validation set while training P2S-SEG on IPSC early-stage dataset. Best viewed under high magnification.

Table 14: Segmentation metrics (

\%

) on validation set while training P2S-VIDSEG on IPSC late-stage dataset. Best viewed under high magnification.

Table 15: Results of our best performing Cityscapes model on the validation set in terms of image-level, class-level and official metrics. Table 16 provides class and category-wise breakdown of the latter. This model was trained with the source images resized from

2048\times 1024

1280\times 640

, followed by sliding window patches of size

P=640

. We used offline RLE with no data augmentation, LAC encoding, frozen backbone,

S=80

L=2K

V=8K

, and

B=80

Table 16: Class and category-wise breakdown of the results of our best performing Cityscapes model on the official cityscapes metrics. Please refer Table 15 for model details. Best viewed under high magnification.

Figure 19: An example of a multi-headed decoder architecture to output each RLE component by a separate head, while sharing the memory-expensive MHA module between them. Please refer to the baseline decoder in [ 17 , Fig. 3] that this adapts. This uses

L=500

S=80

C=2

and naïve RLE encoding with 2D starts and separate length and class tokens. There are thus 4 tokens per run, represented by SX, SY, LEN and CLS, and shown respectively in blue, magenta, green and cyan. Compared to 1D starts and LAC with a single-headed decoder, this not only cuts

L

by half, but also reduces

V

by up to 3 orders of magnitude, since each component now has its own vocabulary which can be as small as

S

for coordinate tokens and

C

for class tokens.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation
Sep 30, 2026Mikhail Dereviannykh, Vikram Voleti, Simon Donne +3Visual Tokenization Autoregressive Video Generation
Stability AI · Karlsruhe Institut für Technologie
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
Sep 30, 2026Yulong Liu, Xiaotian Han, Junyuan Shang +6Video Understanding Efficient VLM Inference
ERNIE Team, Baidu Inc. · The Hong Kong University of Science and Technology · Institute of Automation, Chinese Academy of Sciences (CASIA)
MoVISA: Multi-Token Reasoning for Video Object Segmentation
Sep 24, 2026Ruining Zhao, Ho Kei Cheng, Alexander G SchwingVideo Object Segmentation Referring Video Object Segmentation
University of Illinois Urbana-Champaign

Sep 30, 2026J/K move · Enter open · S save

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy+2

Stability AI · Karlsruhe Institut für Technologie

Visual Tokenization Autoregressive Video Generation Video Token Compression

Open paper AskPDF arXiv

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model

3.4\times

its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.

Abstract

Semantic Segmentation

Explore similar work

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding

MoVISA: Multi-Token Reasoning for Video Object Segmentation