Streaming multi-reference video generation establishes a scene from specified people and objects, then continues it under changing instructions. We introduce TaoMate-R1, a streaming multi-reference generator that organizes adaptation parameters through temporal role specialization. It assigns separate low-rank adapters to scene establishment and continuation on a frozen backbone. Each adapter performs both denoising and context writing for its role; Self Gradient Forcing jointly trains the roles through the history they construct. This allocation contrasts with operation-based specialization, which separates denoising from context writing across all chunks. A fixed-replay gradient analysis characterizes the update directions admitted by the two sharing schemes. On 32 multi-reference cases, we compare the schemes at the same two-adapter budget. Across ten approximately two-minute streams with changing instructions and shared teacher-audio conditioning, TaoMate-R1 reduces the mean boundary luminance jump from 1.96 to 0.613, with higher boundary feature similarity. It also achieves higher mean CLIP and DINO temporal similarity across short interactions, long streams, and public reference compositions. These results support grouping adaptation parameters by scene establishment and continuation for streaming multi-reference generation. Our code and model are released at https://github.com/TaoLiveAIGC/TaoMate-H3.
Figures & tables
Figure 1: Streaming multi-reference video generation. (a) Multiple references guide scene establishment and continuation under changing instructions (conceptual illustration). (b) Each colored group shares one adapter: SGF+ groups computations by operation; TaoMate-R1 groups them by temporal role, sharing generation and context-writing parameters within each role.
Figure 2: TaoMate-R1: generation and context writing through temporal role specialization. The absolute chunk index selects an adapter for both denoising and clean-context writing. The resulting audiovisual keys and values condition subsequent chunks; video latents are decoded for display. The reference images and four stacked output frames come from an evaluated product-interaction sequence.
Method
Reference similarity
Text
Temporal similarity
VABench
CLIP ↑
DINO ↑
CLIP ↑
CLIP ↑
DINO ↑
ViCLIP ↑
Realism ↑
QA ↑
MiniMax-H3
0.660
0.355
0.309
0.934
0.852
0.218
4.50
0.825
VACE
0.680
0.390
0.323
0.946
0.893
0.217
4.59
0.906
SkyReels-V3
0.664
0.372
0.313
0.919
0.842
0.208
4.09
0.575
Phantom
0.664
0.354
0.321
0.927
0.866
0.194
4.19
0.563
Shared
0.656
0.385
0.306
0.951
0.914
0.215
4.63
0.925
Table 1: Multi-reference generation on 32 cases. The upper block gives external systems; the lower block compares adaptation assignments on the same autoregressive backbone. TaoMate-R1 uses Temporal split. Values are case-wise means; bold indicates the best displayed mean within each block. Long-case external outputs concatenate instruction clips; autoregressive variants retain history.
Generator
DINO ref
CLIP temp
DINO temp
QA
Short (10 cases)
Shared
0.525
0.967
0.962
0.820
Operation split
0.531
0.969
0.972
0.900
TaoMate-R1
0.545
0.970
0.975
0.880
Long (10 cases)
Shared
0.469
0.943
0.891
1.00
Table 2: Specialization across interaction settings. Short single-instruction clips, long changing-instruction streams, and public reference compositions. TaoMate-R1 uses Temporal split. Scores are case-wise means; higher is better. Bold indicates the best displayed mean within each group.
Figure 3: Background color and appearance over a long stream. TaoMate-R1 uses the Temporal split configuration. Fixed foliage and vase crops from case 008, enlarged at 0, 60, and 119 s. Below each triplet, the overlay shows mean CIE Lab distance from the initial frame over 119 samples at 1 Hz. All methods use identical crop coordinates and a shared scale (saturated at 12; opacity 0.45).
Figure 4: Interaction sequences under three parameter-sharing schemes. TaoMate-R1 uses Temporal split. Original references are at left; each column shows the same timestamp across all three H3 configurations. (a) Bag presentation and the start of a screen-right set-aside action (case 008). (b) Successive notebook and folded-umbrella presentations (case 006). TaoMate-R1 changes the held object and interaction state while retaining the presenter and room; Operation split remains empty-handed in the sampled umbrella segment.
Generator
CLIP ↑
DINO ↑
Luma ↓
Shared
0.979±0.00737
0.978±0.00585
1.86±0.626
Operation split
0.979±0.00751
0.973±0.0160
1.96±0.456
TaoMate-R1
0.991±0.00268
0.990±0.00412
0.613±0.103
Table 3: Instruction-boundary continuity. Ten long sequences; each cell shows the mean above ± population standard deviation. TaoMate-R1 uses Temporal split. CLIP/DINO compare adjacent boundary frames; Luma is the absolute change in mean luminance. Bold indicates the best mean.
Table 4: Minute-scale continuity. Equal-case means over ten approximately two-minute sequences. DINO temp uses successive 1 Hz samples; DINO bnd and Luma compare adjacent segment-boundary frames.
Figure 6: Scene continuity over a two-minute interaction. Each row combines the first frame’s left half, centre columns sampled every 0.5 s, and the last frame’s right half (case 006, 119 s). Grey ticks mark instruction changes; endpoints retain native aspect ratios. The bands contrast composition changes in clip-wise generation with stable presenter and table geometry in the TaoMate-R1 stream.
Injecting new subjects naturally during streaming video generation requires integrating newly supplied visual references into the existing video while preserving established subjects and the scene. Encoding references independently of the current scene provides little guidance for this integration, while keeping their features in persistent memory can interfere with established context. We propose ReferStream, a framework that requires no additional training and combines reference construction using the current scene with transient cache injection. Joint Latent places reference latents in target regions of the latest scene latent, allowing reference features to be extracted in context. Cache Restoration uses these features to condition the injection chunk, then restores the overwritten global cache entries while retaining the newly generated history. This procedure incorporates new visual conditions without retraining the generator or revising emitted frames. We further introduce ReferStream Bench, a new benchmark with scheduled reference arrivals, evolving instructions, and explicit annotations of subject presence for InShot and MultiShot generation. Evaluation follows the event schedule and measures both the fidelity of new subjects and the preservation of established subjects, alongside video quality, text adherence, and count accuracy. Experiments on ReferStream Bench show higher scores for text adherence, subject similarity, and count accuracy than the compared baselines, including offline methods given all references in advance. Project page: https://xushuolin.github.io/ReferStream/.
Shuolin Xu, Guanyuan Li, Boyu Li +5
Bournemouth University, NCCA, UK · National University of Singapore · Zhejiang University +2
We introduce TokenStream, an optimization-based framework that composes independently learned subject identities in text-to-video generation without a reference-video training dataset or joint optimization of each new subject combination. Learning from a single reference image makes subject appearance difficult to separate from static-image bias, while merging independently trained adapters can mix identities. Token-wise LoRA (ToRA) restricts low-rank updates to the cross-attention value projections of subject-associated text tokens, providing a separate adaptation path for each subject. A T2V-guided stream copies queries and keys from the frozen base model during early denoising to reuse its layout and motion priors while retaining personalized values. We also introduce Multi-DB, a 30-subject, 122-prompt benchmark with appearance-free prompts and varied interactions. On Multi-DB at 1.3B, TokenStream improves identity preservation from 0.422 for DreamBooth to 0.467 and textual alignment from 0.652 to 0.688.
Hung Nguyen, Quang Qui-Vinh Nguyen, Khoi Nguyen
National University of Singapore, Singapore · Qualcomm AI Research
While Test-Time Scaling (TTS) offers a promising direction to enhance video generation without the surging costs of training, current test-time video generation methods based on diffusion models suffer from exorbitant candidate exploration costs and lack temporal guidance. To address these structural bottlenecks, we propose shifting the focus to streaming video generation. We identify that its chunk-level synthesis and few denoising steps are intrinsically suited for TTS, significantly lowering computational overhead while enabling fine-grained temporal control. Driven by this insight, we introduced Stream-T1, a pioneering comprehensive TTS framework exclusively tailored for streaming video generation. Specifically, Stream-T1 is composed of three units: (1) Stream -Scaled Noise Propagation, which actively refines the initial latent noise of the generating chunk using historically proven, high-quality previous chunk noise, effectively establishes temporal dependency and utilizing the historical Gaussian prior to guide the current generation; (2) Stream -Scaled Reward Pruning, which comprehensively evaluates generated candidates to strike an optimal balance between local spatial aesthetics and global temporal coherence by integrating immediate short-term assessments with sliding-window-based long-term evaluations; (3) Stream-Scaled Memory Sinking, which dynamically routes the context evicted from KV-cache into distinct updating pathways guided by the reward feedback, ensuring that previously generated visual information effectively anchors and guides the subsequent video stream. Evaluated on both 5s and 30s comprehensive video benchmarks, Stream-T1 demonstrates profound superiority, significantly improving temporal consistency, motion smoothness, and frame-level visual quality.
Yijing Tu, Shaojin Wu, Mengqi Huang +4
University of Science and Technology of China · Independent Researcher