SPRITETOMESH: Automatic Mesh Generation for 2D Skeletal Animation Using Learned Segmentation and Contour-Aware Vertex Placement

Bastien Gimbert

cs.CVFeb 24, 2026

SPRITETOMESH: Automatic Mesh Generation for 2D Skeletal Animation Using Learned Segmentation and Contour-Aware Vertex Placement

Authors: Bastien Gimbert

Abstract

We present SPRITETOMESH, a fully automatic pipeline for converting 2D game sprite images into triangle meshes compatible with skeletal animation frameworks such as Spine2D. Creating animation-ready meshes is traditionally a tedious manual process requiring artists to carefully place vertices along visual boundaries, a task that typically takes 15-60 minutes per sprite. Our method addresses this through a hybrid learned-algorithmic approach. A segmentation network (EfficientNet-B0 encoder with U-Net decoder) trained on over 100,000 sprite-mask pairs from 172 games achieves an IoU of 0.87, providing accurate binary masks from arbitrary input images. From these masks, we extract exterior contour vertices using Douglas-Peucker simplification with adaptive arc subdivision, and interior vertices along visual boundaries detected via bilateral-filtered multi-channel Canny edge detection with contour-following placement. Delaunay triangulation with mask-based centroid filtering produces the final mesh. Through controlled experiments, we demonstrate that direct vertex position prediction via neural network heatmap regression is fundamentally not viable for this task: the heatmap decoder consistently fails to converge (loss plateau at 0.061) while the segmentation decoder trains normally under identical conditions. We attribute this to the inherently artistic nature of vertex placement - the same sprite can be meshed validly in many different ways. This negative result validates our hybrid design: learned segmentation where ground truth is unambiguous, algorithmic placement where domain heuristics are appropriate. The complete pipeline processes a sprite in under 3 seconds, representing a speedup of 300x-1200x over manual creation. We release our trained model to the game development community.

Figures & tables

Figure 1: Overview of the SpriteToMesh pipeline in six panels. (1) Input sprite image. (2) Binary segmentation mask obtained from the alpha channel or our EfficientNet-B0 network. (3) Exterior contour vertices (blue) placed via Douglas-Peucker simplification with adaptive arc subdivision. (4) Interior boundary vertices (yellow) detected via multi-channel Canny edge detection, combined with exterior vertices (blue). (5) Delaunay triangulation wireframe with centroid-based filtering. (6) Final mesh overlaid on the original sprite.

Figure 2: Exterior contour extraction process in four panels. (a) Binary segmentation mask. (b) Raw contour extracted with findContours , showing thousands of contour pixels. (c) Douglas-Peucker simplification ( $\epsilon=0.003\times P$ ) identifies structural keypoints at corners and concavities (red dots). (d) Final hull vertices (red dots with white outlines) after adaptive arc subdivision adds uniformly-spaced vertices between keypoints along the actual contour path.

Figure 3: Interior vertex placement pipeline shown as a $2\times 3$ grid. (a) Original sprite. (b) Bilateral-filtered image (texture noise reduced, structural edges preserved). (c) Multi-channel Canny edge map (union of per-channel detections). (d) Masked edges after erosion and morphological closing. (e) Interior vertices (yellow) placed along surviving contours using Douglas-Peucker simplification and uniform subdivision. (f) All vertices combined: exterior (blue) and interior (yellow), ready for Delaunay triangulation.

Figure 4: Example of binary-to-JSON conversion using SkelToJson. The binary .skel format encodes the complete skeleton as a dense byte stream (left). Our converter produces structured JSON (right) from which mesh vertex positions, UV coordinates, and triangle indices can be directly extracted.

Figure 5: Dataset preparation pipeline shown in four panels. (1) A full texture atlas from a game. (2) An individual sprite attachment cropped from the atlas using region definitions parsed from the .atlas file. (3) The binary mask extracted from the sprite’s alpha channel ( $\alpha>128$ ). (4) The sprite composited onto a random gradient background, as used during training to force the network to learn foreground–background separation from visual features rather than trivial alpha thresholding.

Figure 6: Representative samples from our dataset, shown as a $4\times 6$ grid where each sprite is drawn from a different game. The dataset spans 172 games and exhibits wide diversity in artistic style, subject matter, complexity, and resolution—including characters, objects, UI elements, effects, and accessories.

Figure 7: Dataset statistics shown as four panels. (a) Histogram of the number of samples per game. (b) Pie chart of attachment type breakdown: region (mask only) vs. mesh (mask + vertex annotations) with percentages and counts. (c) Distribution of vertex counts for mesh samples, with a median line. (d) Distribution of max image dimension (in pixels) across all samples, with a median line.

Figure 8: Data augmentation via background composition, shown as five horizontal panels. From left to right: the original RGBA sprite on a checkerboard (showing transparency), then the same sprite composited onto a solid color, a gradient, and a noise pattern. The rightmost panel shows the ground truth binary mask derived from the alpha channel. This augmentation forces the segmentation network to learn foreground–background separation from visual features.

Metric	Value
IoU	0.8697
Dice Score	0.9304
Pixel Accuracy	0.9741
BCE Loss	0.0523
Dice Loss	0.0696

Table 1: Segmentation performance on the validation set.

Figure 9: Training dynamics shown on a single dual-axis plot. The left axis displays the segmentation loss (BCE + Dice) for both training and validation sets, which decrease steadily with a minimal overfitting gap. The right axis shows the validation IoU, which increases during training and reaches its best value, after which overfitting begins.

Figure 10: Detailed training loss decomposition shown as a $2\times 3$ grid (five panels, one empty). Each panel plots training and validation curves for a single loss component: total loss, segmentation loss (BCE + Dice combined), BCE loss, Dice loss, and heatmap MSE loss. The segmentation-related losses decrease steadily on both sets, while the heatmap loss plateaus early, confirming the infeasibility of direct vertex prediction.

Figure 11: Mesh prediction results on eight diverse sprites arranged in a $2\times 4$ grid. Each cell shows a sprite with the automatically generated mesh overlaid as a wireframe, demonstrating that our pipeline handles a variety of art styles, shapes, and complexities.

	Segmentation	Heatmap
Final loss	0.122	0.061 (plateau)
IoU / Peak recall	0.87	$\approx$ 0
Converged?	✓	$\times$
Prediction quality	Accurate masks	Uniform blur

Table 2: Comparison of segmentation vs. heatmap decoder convergence. The heatmap head plateaus without producing meaningful predictions.

Figure 12: Training loss comparison between segmentation and heatmap decoder heads. Four curves are shown: segmentation loss (green, train and val) and heatmap loss (orange, train and val). The segmentation loss decreases steadily throughout training, while the heatmap loss plateaus at approximately 0.061 after the first few epochs and never improves, demonstrating the fundamental non-learnability of vertex positions from pixel values alone.

Figure 13: Heatmap decoder failure analysis. The heatmap MSE (left axis) remains elevated throughout training with no convergence trend, while the peak recall metric (right axis) drops from an initial value near 1.0, indicating the network’s inability to localize vertex positions. This contrasts sharply with the segmentation decoder’s steady improvement under identical training conditions.

Metric	Value
Exterior vertices	$64\pm 12$
Interior vertices	$420\pm 180$
Total vertices	$484\pm 185$
Triangles	$890\pm 350$
Processing time	$1.8\pm 0.6$ s
Mask coverage	$>99.5$ %

Table 3: Mesh generation statistics across test sprites. Values reported as mean

\pm

std.

Figure 14: Qualitative results on four diverse test sprites, arranged with one sprite per row. Each row shows four panels from left to right: input sprite image, predicted binary segmentation mask, detected vertices (blue = exterior contour, yellow = interior boundary), and final triangulated mesh overlaid on the sprite. Interior vertices are placed along detected visual boundaries within the sprite.

Configuration	Boundary	Vertices	Angle
	Adherence	Count	Ratio
Full pipeline	78.3%	484	0.42
No bilateral filter	71.2%	623	0.38
Single-channel Canny	64.8%	412	0.41
No DP simplification	76.5%	1,847	0.31
No arc subdivision	73.1%	389	0.40

Table 4: Ablation study. We disable individual components and measure their impact on mesh quality.

Figure 15: Comparison of four mesh generation approaches on the same sprite, shown as a $2\times 4$ grid. Top row: vertex placement (blue = exterior, yellow = interior) with vertex counts. Bottom row: resulting triangulated mesh wireframe with triangle counts. (a) Hull only—exterior contour vertices with no interior, yielding a single deformable region. (b) Uniform grid—regular interior vertices that ignore visual structure. (c) Shi-Tomasi corners—feature points that do not follow edge contours. (d) Our contour-aware method—vertices placed along internal visual boundaries, enabling independent deformation of visually distinct regions.

Method	Boundary	Vertices	Articulable
	Adherence		Regions
Alpha-only hull	0%	64	1
Uniform grid	42.6%	520	Partial
Shi-Tomasi corners	55.3%	667	Partial
SpriteToMesh (ours)	78.3%	484	Full

Table 5: Comparison with baseline methods.

Figure 16: Comparison of artist-created and automatically generated meshes, shown as three columns per row: input sprite (left), ground truth artist mesh in green (center), and our predicted mesh in orange with blue exterior and yellow interior vertices (right). While the exact vertex positions differ—reflecting the subjective nature of the task—both meshes cover similar visual boundaries. Our method tends to produce denser meshes along detected edges, while artists place vertices more selectively based on anticipated animation needs.

Figure 17: Examples of current limitations, shown as three mesh prediction results side by side with vertex counts. (a) A highly textured sprite that produces excessive interior vertices due to many spurious Canny edges. (b) A sprite with subtle visual boundaries (soft color gradients) where edges fall below the Canny threshold and are missed. (c) A sprite with unusual proportions or an art style underrepresented in the training data, leading to imperfect segmentation or vertex placement.

Figure 18: Gallery of automatically generated meshes arranged in a $4\times 5$ grid. Each cell shows a single sprite with the predicted mesh overlaid as a wireframe. Our pipeline handles characters, objects, effects, and UI elements from diverse games without any manual intervention or parameter tuning.

Jul 28, 2026

Meshy T2: Fast Native Mesh Generation with Flow Matching

Polygonal meshes are the standard surface representation of modern 3D pipelines, and generating high-quality meshes with artist-style topology is essential for film, gaming, and interactive 3D applications. Mainstream approaches serialize a mesh into a token sequence and decode it autoregressively, which is slow at inference and sensitive to error accumulation, making them impractical for interactive asset creation. We present Meshy T2, a fast native mesh generation framework built on flow matching. At its core is a vertex-set mesh VAE that encodes a mesh into one continuous latent token per vertex and decodes vertices, edge connectivity, and face winding order in a single pass, preserving high-precision geometry and artist-authored topology without vertex quantization or welding. Generation proceeds as a coarse-to-fine cascade of two flow-matching models: an image-conditioned voxel flow first sketches the overall shape as a coarse occupancy scaffold, and a mesh flow then populates the scaffold with per-vertex latent tokens, conditioned on the image, the scaffold, and a requested vertex budget. This design delivers three practical capabilities: interactive generation speed through parallel flow-based synthesis; effective face-count control through the requested vertex budget; and native support for multi-part assets, whose components emerge directly from the generated connectivity. In our experiments, Meshy T2 achieves state-of-the-art geometric fidelity and completes end-to-end image-to-mesh generation within a median of 6 seconds, over an order of magnitude faster than autoregressive baselines. Code and weights will be available at https://github.com/meshy-dev/meshy-t2.

Jiale Xu, Rendong Liang, Yuhao Long +5

Meshy AI

Variational Autoencoders 3D Mesh Generation Flow-Based Generative Modeling

Ask PDF

Apr 10, 2026

Strips as Tokens: Artist Mesh Generation with Native UV Segmentation

Recent advancements in autoregressive transformers have demonstrated remarkable potential for generating artist-quality meshes. However, the token ordering strategies employed by existing methods typically fail to meet professional artist standards, where coordinate-based sorting yields inefficiently long sequences, and patch-based heuristics disrupt the continuous edge flow and structural regularity essential for high-quality modeling. To address these limitations, we propose Strips as Tokens (SATO), a novel framework with a token ordering strategy inspired by triangle strips. By constructing the sequence as a connected chain of faces that explicitly encodes UV boundaries, our method naturally preserves the organized edge flow and semantic layout characteristic of artist-created meshes. A key advantage of this formulation is its unified representation, enabling the same token sequence to be decoded into either a triangle or quadrilateral mesh. This flexibility facilitates joint training on both data types: large-scale triangle data provides fundamental structural priors, while high-quality quad data enhances the geometric regularity of the outputs. Extensive experiments demonstrate that SATO consistently outperforms prior methods in terms of geometric quality, structural coherence, and UV segmentation. Project page: https://ruixu.me/html/SATO/index.html

Rui Xu, Dafei Qin, Kaichun Qiao +8

The University of Hong Kong, Deemos Technology Co., Ltd. , China · ShanghaiTech University, Deemos Technology Co., Ltd., China · Shandong University, China +3

3D Mesh Generation

Ask PDF

Jun 22, 2026

MeshFlow: Mesh Generation with Equivariant Flow Matching

Meshes are among the most common 3D scene representations, but directly generating meshes is challenging because the representation contains important symmetries, including permutation invariance of faces and vertices. MeshFlow learns to generate triangle meshes directly as triangle soups, avoiding the need to serialize meshes into long autoregressive sequences. We adopt equivariant optimal-transport flow matching models that respect the key symmetries of triangle soups: arbitrary permutations of faces and permutations of the vertices within each face. Toward this goal, we propose a simple yet effective modification to the Diffusion Transformer architecture, resulting in a scalable network capable of modeling a velocity field while maintaining the desired equivariance. We further introduce an optimal-transport-based training objective that improves convergence by eliminating supervision signals that violate these symmetries. MeshFlow achieves mesh quality comparable to state-of-the-art autoregressive mesh generators while providing about an 18

\times

speedup during inference. Project page is at https://qiisun.github.io/MeshFlow/.

Qi Sun, Kiyohiro Nakayama, Jing Nathan Yan +6

City University of Hong Kong, Hong Kong · Stanford University, USA · Cornell Tech, USA +1

Flow Matching 3D Mesh Generation Diffusion Transformer Equivariant Neural Networks Flow-Based Generative Modeling

Ask PDF

Abstract

Explore similar work

Meshy T2: Fast Native Mesh Generation with Flow Matching

Strips as Tokens: Artist Mesh Generation with Native UV Segmentation

MeshFlow: Mesh Generation with Equivariant Flow Matching