Texture Interpolation as Texton Matching

Ido Madar; Gal Oren; Shai Avidan

cs.CVOct 13, 2026

Texture Interpolation as Texton Matching

Authors: Ido Madar, Gal Oren, Shai Avidan

Organizations: Tel Aviv University · Stanford University

Abstract

Texture interpolation asks what should lie between two textures. Standard methods typically blend pixels or features at corresponding spatial locations, implicitly assuming that co-located elements should be mixed. For unrelated textures, however, this correspondence is arbitrary: a crack at one location has no particular relationship to a fiber at the same location in another image. We study whether this hidden pairing choice is itself a major determinant of interpolation quality. We represent textures as collections of local appearance vectors, or textons, and replace positional pairing with appearance-aware matching while keeping the encoder, representation, and decoder fixed. This isolates the effect of correspondence from changes in model capacity or training. Across texture pairs, appearance-aware matching consistently improves the distributional quality of interpolated midpoints over positional feature blending. Soft matching produces the strongest midpoint statistics, whereas one-to-one assignments yield more balanced transitions and closer agreement with the target endpoint. The same intervention also improves a second pretrained texture interpolator without retraining. These results identify correspondence as a first-class design choice in texture interpolation: meaningful transitions depend not only on how strongly two textures are mixed, but also on which local appearances are mixed with one another.

Figures & tables

Figure 1: Texture interpolation is a choice of partners. Blending the same locations in unrelated textures can leave two patterns overlaid. We instead associate local appearances, keeping the learned representation and decoder fixed. (a) CNT splats localized Gaussian primitives into dense fields of local appearance vectors (textons). (b) Balanced entropic transport softly matches their vectors by appearance. (c) Each source vector moves toward the weighted mean of its matches. One partner map is reused across $t$ , with source conditioning retained and one decoder pass per frame. (d) Cracked earth to lined fur through RGB cross-fading, positional CNT blending, and soft texton matching. The two CNT rows differ only in pairing. Across experiments, soft matching improves midpoint statistics without retraining, also in Texture Mixer, while one-to-one matching improves midpoint balance and target agreement. Our prompted comparator scores better at midpoints, but has no shared explicit path. The soft texton matching path is directed and need not reconstruct the target. Panels (a–c) are schematic.

Method	KID $\downarrow$	FID $\downarrow$	Inc. real. $\uparrow$	$b$ $\downarrow$	$\Delta$ KID vs. positional
CNT, 1,000 pairs
Positional blending (field-linear)	0.030	108.0	0.925	0.048	–
Texton matching: Gaussian moments	0.024	97.9	0.926	0.105	20%
Texton matching: Soft OT (ours)	0.021	94.7	0.942	0.081	30%
Texton matching: Exact OT	0.033	112.8	0.951	0.044	$-$ 10%
Texture Mixer, 1,000 pairs

Table 1: Does replacing positional blending by texton matching improve interpolation in a fixed decoder? Primary comparison: within each block, encoder, representation and decoder are fixed and the pairing of textons changes; the rendering-controlled comparison, with paired intervals, is Table 2 (CNT, 200 pairs) and Section D.5 (Texture Mixer, 1,000 pairs); exact OT trades set-level statistics for balance. –: not defined for the reference row.

Figure 2: Texton matching within the CNT decoder: rows 2, 4, and 5 of the six-pair grid in Appendix E , Texture Mixer column omitted. Columns: $A$ , CNT-linear, Gaussian $W_{2}$ , Soft OT (ours), $B$ . Distribution alignment blends appearance where the linear baseline retains endpoint structures; Gaussian and empirical transport often look similar.

Rule	KID $\downarrow$	FID $\downarrow$	$b$ $\downarrow$	Inc. real. $\uparrow$
Field-linear (same pipeline)	0.0322	189.3	0.046	0.923
Exact OT (1,024)	0.0300	183.4	0.043	0.952
Soft OT 1,024 $\varepsilon=0.005$	0.0197	174.5	0.067	0.933
Soft OT 1,024 $\varepsilon=0.05$	0.0216	177.6	0.072	0.931
Soft OT 4,096 (ours)	0.0198	174.9	0.076	0.942

Table 2: Is the gain the matching itself, and which form of matching? One rendering path and one scoring session; the exact and 1,024-sample soft rows share the same sampled textons. Soft OT lowers KID by 38% against positional blending; exact OT does not show a statistically resolved improvement over it. Intervals and target-end gaps in Section D.4 .

Method	FID $\downarrow$	KID $\downarrow$	Inc. real. $\uparrow$	DINO real. $\uparrow$	$b$ $\downarrow$	s / midpoint	training
Field-linear	218.9	0.0305	0.913	0.856	0.041	0.18	none
Soft OT (ours)	208.2	0.0190	0.924	0.851	0.081	0.21	none
Exact OT	211.8	0.0302	0.943	0.871	0.044	0.66	none
Texture Mixer	204.0	0.0119	0.948	0.907	0.061	0.012	GAN, domain-specific
Texture Mixer + soft OT	204.0	0.0099	0.942	0.922	0.072	0.012+0.03	GAN, domain-specific
GPT directed	193.7	0.0051	0.960	0.906	0.054	38 (API)	proprietary

Table 3: How does texton matching compare with learned and prompted generators, and at what cost? Prompted generation gives the best midpoint statistics at about 38 s per output; texton matching needs no training and pays its cost once per pair ( Section D.10 ).

Figure 3: The first three pairs of the 120-pair comparison, in pair-ID order. Columns: $A$ , Soft OT (ours), GPT directed, GPT symmetric, Texture Mixer, $B$ . The prompted outputs treat the source arrangement differently from pair to pair; all eight pairs are in Figure 6 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	FID $\downarrow$	KID $\downarrow$	Inc. real. $\uparrow$	DINO real. $\uparrow$	Balance $\downarrow$
Field-linear	218.9	0.0305	0.913	0.856	0.041
Primitive-linear	247.9	0.0504	0.897	0.849	0.040
Gaussian $W_{2}$	216.2	0.0253	0.919	0.847	0.103
Soft OT (ours)	208.2	0.0190	0.924	0.851	0.081
Exact OT	211.8	0.0302	0.943	0.871	0.044
Texture Mixer	204.0	0.0119	0.948	0.907	0.061

Appendix

Table 4: Complete 120-pair results. FID is descriptive at this sample count. Bold marks the best value in each column.

Figure 6: The first eight pairs, in pair-ID order. Columns are $A$ , Soft OT (ours), GPT directed, GPT symmetric, Texture Mixer, and $B$ ; the main text shows the first three rows. The directed prompt handles stochastic appearance, structured motifs, and semantic source content differently; the examples do not establish a pixelwise structure guarantee for either method.

Method	FID $\downarrow$	Midpoint (s)	11-frame path (s)
Soft OT (ours)	268.3	0.21	1.02
Field-linear	284.6	0.18	0.99
Vacher, Gram	318.4	13.5	148.5
Vacher, Gaussian	314.3	15.0	165.3
Vacher, Wasserstein	313.6	52.4	576.6
CNT reconstruction	273.9	–	–

Appendix

Table 5: The 50-pair comparison with optimization-based interpolation, scored at 128 px. Timing is on one A100, with 30 iterations for Vacher variants. FID at this sample count is descriptive. –: not defined, no interpolation is performed.

Layout	Position cost	Appearance cost	Mixed cost
Frozen (source)	0.030	0.030	0.027
Moving (matched)	0.036	0.080	0.046

Appendix

Table 6: KID for one-to-one matchings of CNT’s Gaussian primitives, 120 pairs, canonical scoring. Rows fix or move the source layout; columns give the pairing cost.

Rule	KID $\downarrow$	FID $\downarrow$	$b$ $\downarrow$	Target gap $\downarrow$	Inc. real. $\uparrow$
Random permutation	0.077	224.3	0.035	1.00	1.035
Target mean	0.025	183.7	0.082	0.91	0.921
Field-linear	0.031	188.7	0.044	0.29	0.923
Primitive-linear	0.043	211.8	0.035	0.00	0.895
Gaussian $W_{2}$	0.022	177.6	0.107	0.81	0.928
Nearest neighbor	0.027	182.4	0.046	0.81	0.951

Appendix

Table 7: Matching rules on the same 200 pairs. Random permutation preserves the sampled marginals but ignores appearance. Mean and nearest-neighbor maps test averaging and target reuse. Lower KID/FID is better; gaps are normalized target-end distances (the positional rows reach the target field by construction, so their gap is decoder error only). Bold marks the best value per column; this table uses the original rendering protocol, unlike the matched-rendering table in the main text. Bottom line: every soft appearance-aware rule beats positional pairing on KID, and entropic transport outperforms exact OT (one-to-one) in the matched-rendering comparison ( Table 2 ), so the form of the matching matters, not appearance information alone.

Representation	Positional blend	Appearance-aware matching
DINO	60%	66%
SAM	57%	71%
CNT primitives	60%	87%
CNT field	67%	99%

Appendix

Table 8: Feature spread retained at the midpoint,

n=40

. The common direction across representations supports the matching mechanism. In a separate experiment decoding DINO features to images, appearance-aware matching improves LPIPS midpoint placement on 37 of the

n=40

pairs.

Method	Mean $R$	Fraction $R\geq 1$	Soft OT wins ( $p$ )
Primitive-linear	.920	14%	65/100 ( $2.6\times 10^{-4}$ )
Field-linear	.952	29%	54/100 (.21)
LPIPS control	.938	25%	62/100 (.017)
Soft OT (ours)	.960	41%	–

Appendix

Table 9: Per-image realism,

n=100

R\geq 1

denotes inclusion in the reference manifold estimate. The paired win column compares soft texton matching to each row; –: not defined for its own row.

Figure 7: Complete regularization sweep for one pair. The comparatively small visual change at the midpoint accompanies much larger changes in the entropic plan and projected feature variance.

Method	Realism drop (self)	Realism drop (fixed)	Self drift
Autoencoder only	.105	–	61.9
Primitive-linear	.024	.025	61.0
Field-linear	.024	.029	61.0
Texture Mixer	$-.019$	.006	59.7
Soft OT (ours)	$-.041$	$-.077$	62.8

Appendix

Table 10: Eight repeated operations on 24 pairs. Self: use the current image on both sides. Fixed: retain a fixed second endpoint. Positive realism drop denotes deterioration under that observer; negative values denote an increase. –: not defined, no second endpoint.

Method	Encode, once per pair	Couple, once per pair	Each requested $t$
Field-linear	0.09	–	0.08 (0.02) decode
Soft OT (ours)	0.09	0.03	0.08 (0.02) decode
Exact OT	0.09	0.48	0.08 (0.02) decode
Texture Mixer	0.009	–	0.003 generate
Vacher $W_{2}$	–	–	52.4 optimize
GPT-Image-1.5	–	–	$\sim 38$ API call

Appendix

Table 11: Recorded stage timings in seconds on one A100: 120 pairs for the CNT and Texture Mixer rows; the Vacher et al. timing comes from a 20-pair run on the same hardware with its 30-iteration configuration; GPT is the median API latency over the scored calls. Encoding is separate; each interpolation coefficient is one renderer pass. CNT decoding is at 256 px (128 px in parentheses); Texture Mixer is at 128 px. –: no such stage for this method.

Figure 8: The full six-pair qualitative comparison. The main paper selects rows 2, 4, and 5. The Gaussian and empirical columns often resemble one another, while the relative appearance of all methods depends on the pair.

Figure 9: Full correspondence progression. Columns: $A$ , positional linear, Soft OT (ours), $B$ . Rows: overlapping crops, a $2^{\circ}$ warp, an $8^{\circ}$ warp, and an unrelated pair. Spatial pairing remains useful in the aligned rows; appearance matching gives a coherent intermediate in the unrelated pair.

Figure 10: Alignment isolated. Rows: cyclic shift of $B$ by 0, 4, 16, 64 px; columns: source reconstruction, field-linear, Gaussian, Soft OT (ours), $B$ . Positional blending superposes as the shift grows; the appearance-aware rules are far less sensitive to displacement.

Figure 11: A second four-texture example: a $2\times 2$ collage of four textures moved toward their anchor-based mixture as $t$ increases.

Figure 12: A $4\times 4$ palette spanned by four corner textures under one anchor; every cell is a decoded mixture with the same conditioning.

Figure 13: Exploratory: a natural image clustered into three regions and moved toward their anchor-based mixture. Objects dissolve as distributions; the construction is not an exact barycenter and is outside the texture setting of the paper.

Figure 14: Exploratory dictionary reconstructions. Columns show the target, CNT reconstruction, and decoded constructions from two dictionary choices. The bottom row illustrates the difficulty of preserving regular structure.

Figure 15: Plan-alignment histograms. The separated low-alignment group appears under sharp and hard plans. The $\rho^{*}$ notation in the histogram refers to this statistic; only exact unregularized optimization maximizes alignment over the sampled coupling set.

Texture Synthesis

Texture synthesis creates and edits realistic visual patterns and surface materials, from image patches to coherent, view-consistent 3D assets.

See all 34 papers

Image Tokenizers Should Be Trained Between Images
Oct 12, 2026Ole Beißwenger, Jan-Niklas Dihlmann, Hendrik LenschImage Tokenization Latent Diffusion Models
University of Tübingen
GLOSS: Geometric Local Self-Similarity Learning for Faithful Reference-Guided Texture Fill
Aug 26, 2026Chenyue Cai, Anita Hu, James Lucas +23D Texture Generation Texture Synthesis
Princeton University, USA · NVIDIA, Canada · NVIDIA, UK +1
Denoising Textures is a Different Ballgame Altogether!
Oct 15, 2026Saptarshi Mandal, Oindrila Haldar, Chirayata Bhattacharya +4Image Denoising
Department of Electrical Engineering, Indian Institute of Science, Bangalore, India · Department of Electrical and Computer Engineering, Northwestern University, Evanston, Illinois, USA · Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA

Oct 12, 2026J/K move · Enter open · S save

Image Tokenizers Should Be Trained Between Images

Ole Beißwenger, Jan-Niklas Dihlmann, Hendrik Lensch

University of Tübingen

Image Tokenization Latent Diffusion Models Diffusion-Based Image Generation

Open paper AskPDF arXiv

Latent diffusion models generate images as codes of an image tokenizer, whose decoder turns them into the pixel image. Recent tokenizers improve how well these codes reconstruct an image and how easily a generator learns them. Yet a generator does not reproduce the codes of training images, and when a generated code lands outside what the decoder expects, the decoded image contains imperfections. Current methods make the decoder more robust to generated codes by adding noise during training. However, generated codes lie between the training codes and resemble their mixture more than one of them plus noise. To improve generated images, decoders should therefore be trained on codes in the space between training samples. We propose InterpTok, which fine-tunes MAETok to decode mixtures of image codes into natural images. Since no target image exists between two training samples, a discriminator judges whether each decoded mixture looks as natural as its two sources. A mixing ratio loss requires the decoded mixture to keep the mixing ratio of the two codes. Together, they prevent collapse to a trivial solution and yield a sharp and natural composition of both images. InterpTok keeps MAETok's reconstruction quality and improves interpolation quality more than threefold, the best of all tokenizers in a recent benchmark. Beyond interpolation, InterpTok improves the generation FID of a matched diffusion transformer from 11.61 to 8.48 without guidance, and even reaches 8.11 at an eighth of the sampling steps, where the generator on MAETok's codes worsens. The generator trained on our tokenizer also reaches the final result of the one trained on MAETok at half the training steps. Our findings show that the space between image codes can and should be explicitly supervised rather than ignored.

Abstract

Texture Synthesis

Explore similar work

Image Tokenizers Should Be Trained Between Images

GLOSS: Geometric Local Self-Similarity Learning for Faithful Reference-Guided Texture Fill

Denoising Textures is a Different Ballgame Altogether!