Iterative diffusion generation requires different amounts of model capacity at different token positions. Prior model combinations allocate capacity by denoising step or by drafting and verification role. We present HybridStitch, a training-free framework that uses token-level feedback to select the model supplying each adopted update. Continuous image refinement and discrete language commitment operate on a shared generation state, while model-specific KV buffers provide attention context for selected queries. HybridStitch achieves 1.63--1.68× faster image generation and 1.15--1.58× diffusion large language model (dLLM) throughput on four main benchmarks. Estimated dLLM decode query cost decreases by 50--66% with comparable generation quality.
Figures & tables
Figure 1: Allocation granularity. (a) Step-level image switching and role-level text speculation. (b) Spatial disagreement and illustrative token confidences. (c) Token-level allocation in HybridStitch .
Figure 2: Small–large agreement on identical inputs. (a) SD3 velocity discrepancy and its 10–90 % token range at 1024×1024 . (b) Text agreement by small-model confidence.
Figure 3: HybridStitch execution framework. (a) Generation feedback produces a routing mask for model-specific token work. (b) Selected queries process attention with cached keys and values.
Dimension
Continuous (T2I)
Discrete (dLLM)
Feedback
Disagreement
Confidence
Selection
Budgeted TopK
Threshold and frontier
Owner πs(i)
Mask-selected candidate
Committing model; ∅ holds
Scheduling
Both models per hybrid step
One model per request call
Shared state
Merged latent
Committed tokens
Context
Model-specific KV and selected queries
Table 1: Two realizations of token ownership.
Task
Method
Score
Tok/s
Speedup
GF/tok
Mean/p90 (s)
GSM8K
Plain SDAR
91.5
308
1.00 ×
179.1
12.1/19.4
dKV adaptation
90.0
333
1.08 ×
127.5
11.1/17.3
SimSD
88.0
320
1.04 ×
121.4
12.6/18.1
HybridStitch
89.5
354
1.15 ×
90.2
13.7/20.8
MBPP
Plain SDAR
54.5
188
1.00 ×
330.1
6.3/9.3
dKV adaptation
51.0
222
1.18 ×
205.1
5.3/7.9
Table 2: dLLM end-to-end results.
Method
FID ↓
CLIP ↑
LPIPS ↓
Lat. ↓
COCO
Large
27.64
31.81
–
13.64
T-Stitch
31.87
32.01
0.70
9.63
SRDiffusion
31.67
31.89
0.73
9.43
HybridStitch
30.43
31.84
0.42
8.11
DiffusionDB
Table 3: SD3 end-to-end results. Latency is seconds per image.
Figure 4: GSM8K and MBPP throughput scaling relative to plain SDAR.
Method
FID
CLIP
LPIPS
Lat.
StaticMask
31.88
31.79
0.45
7.72
FullLarge
31.32
31.84
0.39
8.97
HybridStitch
30.43
31.84
0.42
8.11
Table 4: SD3 ablations on COCO. StaticMask freezes routing; FullLarge computes all large-model queries but merges selected outputs.
Figure 5: Small-model ownership by block position.
Figure 6: T2I ownership-budget transitions (top) and dLLM committed owners by block iteration (bottom).
Figure 7: Quality–cost sweeps with fitted trends: (a) T2I, (b) GSM8K, and (c) MATH500.