We introduce StreamEval with four key features: (1) First interactive streaming video generation benchmark. (2) Fully open source. Benchmark data, evaluation code, protocols, and scoring rules are publicly available. (3) Comprehensive coverage. Five tasks cover avatar dialogue, avatar performance, world generation, instruction editing, and reference editing, with 300 scripted cases lasting 15 to 300 seconds. Ten metrics assess task execution, state consistency, visual fidelity, and audiovisual coordination. (4) Monthly updates. We will evaluate new and updated models and publish leaderboard results monthly. Results from 37 models show that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency. The leaderboard is available at https://streameval.ai.
Figures & tables
Figure 1: StreamEval overview. Five tasks span avatar interaction, world generation, and real-time video editing. Avatar panels show recorded frames; the world-generation panel illustrates a scenario. Editing uses a shared source video with text instructions or reference images that change over time.
Figure 2: Benchmark construction and verification. Model-assisted authoring and input synthesis precede modality-specific checks. A Qwen-Image portrait with hand defects is regenerated and passes the same screening.
Figure 3: Benchmark composition. (a) Task and function composition. (b) Planned input types. (c) Inputs per session. (d) Gaps between adjacent scheduled inputs. The 200 cases shown contain 562 online events, excluding initialization and continuous source frames. Gap statistics exclude pairs containing a trigger.
Benchmark
Task coverage
Duration (s)
Dlg.
Perf.
World
Instr.
Ref.
VBench [ 21 ]
×
×
✓
×
×
2–3.3
VBench++ [ 22 ]
×
×
✓
×
×
Var.
T2V-CompBench [ 51 ]
×
×
✓
×
×
2–5
TC-Bench [ 11 ]
×
×
✓
×
×
Var.
NarraStream-Bench [ 38 ]
×
×
✓
×
×
60
Table 1: Five-task coverage and evaluation duration. Dlg.: dialogue; Perf.: avatar performance; World: world generation; Instr./Ref.: instruction/reference real-time video editing. ✓ / × denote task inclusion/exclusion.
Figure 4: Evaluation workflow. Recorded inputs and output frames from reference-editing case 234 illustrate quality, stability, reference, and preservation checks. Fixed rules convert structured VLM observations into run metric scores. Aggregation averages repeats, conditions, and duration strata before computing task-specific Overall.
Model
Quality
A–V Sync
Stability
Dialogue
Identity
Q-Overall
Custom.
Overall
AptAvatar 14B †
68.03
70.32
94.74
35.98
81.00
70.01
+5
75.01
Avatar-Forever 22B †
50.86
48.17
68.48
54.93
75.23
59.53
+5
64.53
AvatarForcing 1.3B †
57.17
0.00
87.30
8.93
66.85
44.05
+5
49.05
AVTR-1 †
82.97
69.38
99.87
54.97
88.54
79.14
+5
84.14
LiveAvatar FULL
76.27
0.00
100.00
61.37
87.70
65.07
0
65.07
LiveTalk 1.3B †
61.75
62.80
86.51
54.40
77.51
68.60
+5
73.60
Table 2: Digital-human dialogue.
Model
Quality
A–V Sync
Stability
Following
Identity
Q-Overall
Custom.
Overall
AptAvatar 14B
55.55
98.89
78.06
13.04
81.33
65.37
+5
70.37
Avatar-Forever 22B
34.73
34.67
45.31
6.37
74.82
39.18
+5
44.18
AvatarForcing 1.3B
54.32
18.44
71.60
8.44
68.50
44.26
+5
49.26
LiveTalk 1.3B
71.27
86.44
96.06
16.59
83.89
70.85
+5
75.85
Quark Live Avatar 14B
43.51
98.89
55.30
11.70
79.75
57.83
+5
62.83
SoulX-LiveAct 18B
48.39
48.67
64.94
10.74
69.83
48.51
+5
53.51
Table 3: Avatar performance.
Model
Quality
Stability
Following
Continuity
Overall
AlayaWorld
49.44
46.17
8.89
10.08
28.65
Causal Forcing
18.12
15.83
15.23
14.11
15.82
Context Forcing
35.70
31.04
18.52
16.49
25.44
HappyOyster 1.0 Adventure
58.34
56.67
5.56
2.78
30.84
Helios
28.79
28.04
14.44
19.90
22.79
HY-World 1.5 WorldPlay
50.60
48.06
10.00
17.11
31.44
Table 4: World generation. Text- and image-conditioned configurations are ranked together; a model evaluated with both inputs is reported with its text-conditioned result.
Model
Quality
Stability
Following
Edit Success
Preservation
Overall
EditaLive (14B)
7.86
4.12
12.04
4.26
23.15
10.29
JoyAI-Video-Edit
11.63
9.57
63.89
43.52
83.33
42.39
Lucy 2.5
19.94
9.40
36.48
12.12
65.00
28.59
SANA-Streaming
15.15
9.22
11.48
4.63
44.26
16.95
Vidu S2 Editing
22.99
13.94
72.10
57.20
60.93
45.43
XMax X2.0
5.14
0.97
11.30
2.59
28.52
9.70
Table 5: Instruction editing.
Model
Quality
Stability
Reference
Edit Success
Preservation
Overall
JoyAI-Video-Edit
6.54
3.46
26.11
8.89
26.67
14.33
Lucy 2.5
25.55
28.55
7.59
1.30
30.38
18.67
Vidu S2 Editing
37.72
46.74
31.20
15.00
64.44
39.02
XMax X2.0
8.20
7.51
20.56
1.11
10.00
9.47
Table 6: Reference editing.
Figure 5: Reference editing under changing inputs. Reference A is preloaded, B arrives at 10 s, and removal is requested at 20 s. Columns pair source and output frames at recorded session times. Insets magnify the outlined cape regions, showing grey retained in both B-phase outputs instead of the requested gold satin.