Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation

Haibo Jin; Xinjie Li; Najmeh Sadoughi; Yang Liu; Yibo Wang; Zhu Liu; Yuzong Liu

cs.CLSep 29, 2026

Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation

Authors: Haibo Jin, Xinjie Li, Najmeh Sadoughi, Yang Liu, Yibo Wang, Zhu Liu, Yuzong Liu

Organizations: University of Illinois Urbana-Champaign, USA · Amazon, USA

Abstract

Long-form subtitle translation requires reasoning over discourse and cultural context spanning episodes or entire series, while maintaining consistent terminology and style. Existing single-LLM methods are largely sentence-level, and multi-agent systems often use static workflows that do not adapt to scene complexity or production context. We propose SMART, a Self-evolving Multi-Agent system for long-foRm subtitle Translation. During test-time training, SMART builds persistent series-level memory and translates a subset of sentences through a dynamic router and Mixture-of-Agents layer with tools for terminology verification, subtitle constraint validation, and contextual retrieval. A judge-refiner loop scores candidates and uses textual critiques to update agent prompts and routing policies without retraining the underlying LLMs. During test-time inference, the evolved configuration translates the remaining series. We also introduce Subtitle Arena, covering 14 genres, 2--198 episodes per series, production years 1959--2023, and 15 target locales, together with SubMQM, a subtitle-adapted MQM framework with seven dimensions and 19 error categories. SMART achieves the best overall MQM score in all 15 Subtitle Arena directions, reducing average penalty by 6.9% over the strongest competing agent system. On a public benchmark, MuSC, SMART obtains the best model result across all 4 language pairs. SMART also achieves the best result in human evaluation with an overall score of 4.50/5.

Figures & tables

Figure 1: Overview of SMART . Given source data in SRT style, SMART parses it into sentences. For each sentence, a learnable router ( Graph Router ) activates a subset of translators ( MoA Translation ); candidates may invoke tools ( Tool-Calling ) and are selected and refined by a judge–refiner loop ( Judge-Refine ). A persistent Memory Pool carries terminology, character information, domain knowledge, and idiomatic expressions across episodes. During test-time training, translation feedback updates translator prompts and routing.

Aligned Episode Coverage by Target Locale
Locale	zh-CN	pt-BR	es-ES	es-MX	fr-FR	ro-RO	tr-TR	pt-PT
Episodes	6,267	5,407	5,144	5,144	5,079	5,002	4,956	4,853
Locale	de-DE	it-IT	nl-NL	sv-SE	da-DK	no-NO	ko-KR	Total
Episodes	4,697	4,518	4,518	4,286	4,273	3,277	3,243	70,664
Series Distribution by Genre
Genre	Comedy	Action	Crime	Animation	Fantasy	Drama	Sci-Fi

Table 1: Statistics of Subtitle Arena . Top: aligned bilingual episode coverage for each target locale, measured against

6{,}267

English source episodes. Bottom: distribution of the

192

television series across

14

genres.

Dimension	Representative Errors	#	Weight
Accuracy	Mistranslation, omission, overtranslation	3	0.30
Terminology	Name and term inconsistency	2	0.20
Fluency	Coherence, naturalness, vividness	3	0.20
Audience Appropriateness	Profanity, formality	2	0.12
Linguistic Conventions	Punctuation, capitalization, grammar, spacing	4	0.08
Technical	Line breaking, CPL, lines per box	3	0.06

Table 2: Overview of SubMQM. Each error type is assigned a penalty in

\{0,5,10\}

; lower is better. The complete 19-type rubric is provided in Appendix E .

Language Direction	Method	Term.	Acc.	Flu.	Ling.	Tech.	Locale	Audience	Overall
en $\rightarrow$ zh $\|$ zh $\rightarrow$ en	Online	0.35 $\|$ 0.42	4.87 $\|$ 4.23	4.07 $\|$ 3.28	1.98 $\|$ 1.47	0.20 $\|$ 0.29	1.40 $\|$ 0.11	0.05 $\|$ 0.22	2.58 $\|$ 2.17
	Gemma 3 4B	0.49 $\|$ 0.52	2.76 $\|$ 3.10	1.97 $\|$ 2.16	0.59 $\|$ 0.81	0.79 $\|$ 1.00	0.15 $\|$ 0.16	0.35 $\|$ 0.39	1.46 $\|$ 1.64
	DeepSeek-V3.2	0.28 $\|$ 0.31	1.86 $\|$ 2.03	1.25 $\|$ 1.43	0.29 $\|$ 0.42	0.49 $\|$ 0.88	0.03 $\|$ 0.08	0.13 $\|$ 0.18	0.93 $\|$ 1.07
	Claude 4.6	0.26 $\|$ 0.27	1.63 $\|$ 1.84	1.16 $\|$ 1.28	0.26 $\|$ 0.34	0.42 $\|$ 0.83	0.03 $\|$ 0.05	0.16 $\|$ 0.15	0.84 $\|$ 0.96
	Claude 4.8	0.25 $\|$ 0.20	1.56 $\|$ 1.39	1.09 $\|$ 1.00	0.20 $\|$ 0.23	0.32 $\|$ 0.68	0.04 $\|$ 0.03	0.13 $\|$ 0.11	0.78 $\|$ 0.73
	GPT-5.5	0.20 $\|$ 0.18	1.16 $\|$ 1.20	1.13 $\|$ 0.98	0.19 $\|$ 0.19	0.22 $\|$ 0.87	0.03 $\|$ 0.03	0.14 $\|$ 0.13	0.66 $\|$ 0.67

Table 3: SubMQM results on representative bidirectional language pairs from Subtitle Arena. Each cell reports both directions in the order shown in the first column (left

|

right). Dimension scores are averaged over their constituent error types. All values are penalties (lower is better). The best and second-best values are highlighted in blue and orange, respectively.

		en $\rightarrow$ zh			ko $\rightarrow$ zh			zh $\rightarrow$ en			zh $\rightarrow$ th
Model	Training	Acc.	Nat.	Viv.	Acc.	Nat.	Viv.	Acc.	Nat.	Viv.	Acc.	Nat.	Viv.
Gold Reference	Human	83.6	82.6	71.5	78.0	77.8	65.8	83.0	80.3	73.3	76.6	75.1	66.3
VideoDubber	–	46.9	51.9	49.7	39.6	45.2	48.2	53.6	54.8	50.1	34.1	34.9	41.5
NLLB-3.3B	–	61.4	54.0	43.7	33.1	26.1	25.4	29.1	21.7	20.8	42.6	33.9	40.5
MADLAD-10B	–	59.7	55.5	46.3	44.9	42.9	46.7	45.1	38.9	37.6	47.9	50.8	51.0
Google Translate	–	84.2	79.7	54.4	54.9	52.8	52.0	79.8	66.3	50.2	55.2	56.2	54.5

Table 4: MuSC results across four directions. Higher is better. Best and second-best are highlighted in blue-gray and warm beige.

Language Direction	Method / Setting	Term.	Acc.	Flu.	Ling.	Tech.	Locale	Audience	Overall
en $\rightarrow$ zh $\|$ zh $\rightarrow$ en	Gemma 3 4B	0.49 $\|$ 0.52	2.76 $\|$ 3.10	1.97 $\|$ 2.16	0.59 $\|$ 0.81	0.79 $\|$ 1.00	0.15 $\|$ 0.16	0.35 $\|$ 0.39	1.46 $\|$ 1.64
	DeepSeek-V3.2	0.28 $\|$ 0.31	1.86 $\|$ 2.03	1.25 $\|$ 1.43	0.29 $\|$ 0.42	0.49 $\|$ 0.88	0.03 $\|$ 0.08	0.13 $\|$ 0.18	0.93 $\|$ 1.07
	SMART (Gemma 3 4B)	0.23 $\|$ 0.19	1.14 $\|$ 1.12	1.27 $\|$ 1.06	0.11 $\|$ 0.16	0.03 $\|$ 0.11	0.07 $\|$ 0.05	0.19 $\|$ 0.12	0.67 $\|$ 0.62
	SMART (DeepSeek-V3.2)	0.18 $\|$ 0.15	0.89 $\|$ 0.92	1.04 $\|$ 0.91	0.08 $\|$ 0.12	0.01 $\|$ 0.09	0.05 $\|$ 0.03	0.14 $\|$ 0.10	0.53 $\|$ 0.51
	SMART (Judge: GPT-5.5)	0.15 $\|$ 0.12	0.80 $\|$ 0.76	0.90 $\|$ 0.82	0.07 $\|$ 0.10	0.00 $\|$ 0.08	0.04 $\|$ 0.02	0.12 $\|$ 0.09	0.47 $\|$ 0.44
	SMART (Claude 4.6)	0.15 $\|$ 0.13	0.80 $\|$ 0.80	0.95 $\|$ 0.79	0.07 $\|$ 0.10	0.00 $\|$ 0.08	0.04 $\|$ 0.02	0.12 $\|$ 0.09	0.48 $\|$ 0.44

Table 5: Backbone and judge analysis on 4 bidirectional pairs from Subtitle Arena. Each cell gives both directions in the first column’s order (left

|

right). All values are penalties (lower is better). Blue-gray and warm-beige mark the best and second-best values.

Setting	en $\rightarrow$ zh	en $\rightarrow$ de	en $\rightarrow$ ko	en $\rightarrow$ it	en $\rightarrow$ es	en $\rightarrow$ fr
Full system	0.48	0.88	0.93	1.13	1.18	1.19
w/o Dynamic Router	0.70 (+0.22)	1.12 (+0.24)	1.18 (+0.25)	1.47 (+0.34)	1.46 (+0.28)	1.48 (+0.29)
w/o Self-Evolution	0.80 (+0.32)	1.22 (+0.34)	1.28 (+0.35)	1.50 (+0.37)	1.55 (+0.37)	1.57 (+0.38)
w/o Memory	1.43 (+0.95)	1.78 (+0.90)	1.69 (+0.76)	1.67 (+0.54)	1.74 (+0.56)	2.13 (+0.94)
w/o Contextual Retrieval & Idiom Bank	1.24 (+0.76)	1.56 (+0.68)	1.31 (+0.38)	1.49 (+0.36)	1.44 (+0.26)	2.08 (+0.89)
w/o Sliding Window	0.93 (+0.45)	1.41 (+0.53)	1.52 (+0.59)	1.52 (+0.39)	1.58 (+0.40)	1.64 (+0.45)

Table 6: Component ablation of SMART across six directions. Values are Overall SubMQM penalties (lower is better); parentheses show increases over the full system.

Method	Fidelity	Consistency	Language	Subtitle	Overall
Online	1.10	1.10	1.05	1.10	1.05
Claude Sonnet 4.6	3.65	3.35	3.80	3.90	3.75
GPT-5.5	2.45	2.30	2.45	2.30	2.30
TransAgent	3.35	3.70	3.30	3.25	3.40
SMART	4.45	4.55	4.40	4.45	4.50

Table 7: Human evaluation by average rank (

5

best,

1

worst). Best and second-best are highlighted in blue-gray and warm beige.

Figure 2: Qualitative examples of context-aware subtitle translation. (a) SMART preserves the wordplay between “humor” and “humerus” through a localized Chinese rendering involving “肱骨”. (b) External web knowledge identifies “Bullets” as the nickname of Lt. Grace Billets, allowing SMART to translate the utterance as a reference to the officer rather than the literal noun “bullets”.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Purpose	Execution
terminology_lookup	Retrieve the established target rendering of a recurring name or term.	Match against the shared terminology table and return the confirmed translation or partial matches.
terminology_register	Fix the target rendering of a new name or term.	Write to the terminology table with conflict detection to prevent overwriting an existing entry.
memory_search	Find previously translated sentences with similar source content.	Rank entries from the current episode’s translation memory by source overlap to support consistency and style.
get_context	Retrieve surrounding dialogue, scene information, and domain notes.	Read up to eight preceding and succeeding sentences together with the associated scene metadata.
constraint_check	Verify subtitle reading-speed and line-length constraints.	Compute characters per second and per-line character counts, and report specific violations.
web_search	Resolve jargon, cultural references, or ambiguous slang.	External search whose raw results are interpreted to separate literal meaning from the register appropriate to the scene.

Appendix

Table 8: Tool modules available to SMART translators. Execution is local unless otherwise noted.

Resource	Unit	Domain	Directions	Series context	Display constraints	Evaluation
MuST-Cinema	Talk	TED talks (speech)	en $\rightarrow$ 7 languages	$\times$	✓	BLEU
BigVideo	Clip	Web video	en $\rightarrow$ zh	$\times$	$\times$	BLEU
OpenSubtitles2024	Sentence	Film and television	Many	$\times$	$\times$	Corpus only
SubScene	Sentence	Film and television	Many	$\times$	$\times$	Corpus only
MuSC	Utterance segment	Streaming programs	6 directions	$\times$	$\times$	Accuracy, naturalness, vividness
Subtitle Arena	Series	Television	30 directions (15 locales, both ways)	✓	✓	SubMQM (7 dimensions, 19 error types)

Appendix

Table 9: Subtitle Arena against existing subtitle translation resources. Unit is the largest span the resource is organized around; Series context indicates whether context persists across episodes of the same title; Display constraints indicates whether subtitle timing and line-length limits are part of the evaluation.

Figure 3: Visualization of Subtitle Arena statistics. Left: aligned bilingual episode coverage for each target locale, measured against $6{,}267$ English source episodes. Right: distribution of the $192$ television series across $14$ genres.

Dimension	Error Type	Description
Terminology	Name Inconsistency	Inconsistent translations or spellings of recurring proper names, such as characters, places, and entities.
	Term Inconsistency	Inconsistent translations of recurring domain-specific or contextual terms and phrases.
Accuracy	Mistranslation	Translation that incorrectly changes the meaning of the source.
	Undertranslation	Source content that should be translated is omitted.
	Overtranslation	Unsupported information or specificity is introduced into the translation.
Fluency	Coherence	The translation lacks continuity with the surrounding scene or discourse.

Appendix

Table 10: Subtitle-adapted MQM rubric used by SubMQM. The rubric contains seven dimensions and 19 error types. Each error type receives a penalty in

\{0,5,10\}

; lower is better.

Dir.	Method	Terminology		Accuracy			Fluency			Overall
		NameInc	TermInc	MisTrans	UndTrans	OvrTrans	Coher	Natur	Vivid
en $\rightarrow$ zh	Online	0.40	0.30	4.40	5.60	4.60	6.30	5.00	0.90	2.58
	Gemma 3 4B	0.52	0.46	3.45	2.68	2.15	2.65	2.12	1.15	1.46
	DeepSeek-V3.2	0.31	0.25	2.52	1.76	1.30	1.88	1.29	0.58	0.93
	Claude 4.6	0.26	0.25	2.31	1.43	1.15	2.03	0.90	0.56	0.84
	Claude 4.8	0.27	0.22	2.13	1.31	1.23	1.40	1.08	0.78	0.78

Appendix

Table 11: Fine-grained SubMQM results for translation from English into 15 target locales: Terminology, Accuracy, and Fluency. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 3 .

Dir.	Method	Linguistic Conventions				Technical			Locale Conventions		Audience Appropriateness		Overall
		MisPunc	MisCap	Gram	Space	LineBrk	CharLim	LineLim	LocErr	LangDet	Profan	Formal
en $\rightarrow$ zh	Online	4.30	0.00	0.50	3.10	0.10	0.50	0.00	0.00	2.80	0.10	0.00	2.58
	Gemma 3 4B	1.05	0.00	0.58	0.72	1.18	0.84	0.35	0.18	0.12	0.38	0.31	1.46
	DeepSeek-V3.2	0.61	0.00	0.19	0.34	0.82	0.52	0.14	0.04	0.01	0.15	0.10	0.93
	Claude 4.6	0.55	0.00	0.16	0.31	0.70	0.44	0.12	0.04	0.02	0.20	0.11	0.84
	Claude 4.8	0.41	0.00	0.13	0.26	0.53	0.34	0.09	0.05	0.02	0.15	0.10	0.78

Appendix

Table 12: Fine-grained SubMQM results for translation from English into 15 target locales: Linguistic Conventions, Technical, Locale Conventions, and Audience Appropriateness. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 3 .

Dir.	Method	Terminology		Accuracy			Fluency			Overall
		NameInc	TermInc	MisTrans	UndTrans	OvrTrans	Coher	Natur	Vivid
zh $\rightarrow$ en	Online	0.47	0.36	4.92	4.05	3.71	4.56	3.79	1.49	2.17
	Gemma 3 4B	0.58	0.45	3.91	2.89	2.51	2.91	2.24	1.34	1.64
	DeepSeek-V3.2	0.36	0.26	2.74	1.86	1.50	2.11	1.39	0.78	1.07
	Claude 4.6	0.32	0.22	2.49	1.68	1.34	1.89	1.26	0.69	0.96
	Claude 4.8	0.24	0.16	1.86	1.24	1.08	1.49	0.98	0.54	0.73

Appendix

Table 13: Fine-grained SubMQM results for translation from 15 source locales into English: Terminology, Accuracy, and Fluency. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 3 .

Dir.	Method	Linguistic Conventions				Technical			Locale Conventions		Audience Appropriateness		Overall
		MisPunc	MisCap	Gram	Space	LineBrk	CharLim	LineLim	LocErr	LangDet	Profan	Formal
zh $\rightarrow$ en	Online	2.16	1.19	1.71	0.81	0.49	0.31	0.06	0.18	0.03	0.28	0.15	2.17
	Gemma 3 4B	1.21	0.56	0.88	0.59	1.51	1.04	0.44	0.26	0.05	0.46	0.31	1.64
	DeepSeek-V3.2	0.64	0.32	0.46	0.24	1.41	0.99	0.24	0.15	0.01	0.25	0.10	1.07
	Claude 4.6	0.59	0.18	0.39	0.18	1.31	0.89	0.29	0.06	0.04	0.15	0.15	0.96
	Claude 4.8	0.42	0.11	0.28	0.12	1.11	0.72	0.22	0.03	0.03	0.11	0.12	0.73

Appendix

Table 14: Fine-grained SubMQM results for translation from 15 source locales into English: Linguistic Conventions, Technical, Locale Conventions, and Audience Appropriateness. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 3 .

Dir.	Method / Setting	Terminology		Accuracy			Fluency			Overall
		NameInc	TermInc	MisTrans	UndTrans	OvrTrans	Coher	Natur	Vivid
en $\rightarrow$ zh	Gemma 3 4B	0.52	0.46	3.45	2.68	2.15	2.65	2.12	1.15	1.46
	DeepSeek-V3.2	0.31	0.25	2.52	1.76	1.30	1.88	1.29	0.58	0.93
	SMART (Gemma 3 4B)	0.30	0.16	1.20	1.08	1.14	1.39	1.28	1.14	0.67
	SMART (DeepSeek-V3.2)	0.22	0.13	0.96	0.88	0.84	1.14	1.00	0.98	0.53
	SMART (Judge: GPT-5.5)	0.18	0.12	0.90	0.83	0.68	1.03	0.89	0.78	0.47

Appendix

Table 15: Fine-grained backbone and judge analysis on Subtitle Arena for translation from English into 15 target locales: Terminology, Accuracy, and Fluency. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 5 .

Dir.	Method / Setting	Linguistic Conventions				Technical			Locale Conventions		Audience Appropriateness		Overall
		MisPunc	MisCap	Gram	Space	LineBrk	CharLim	LineLim	LocErr	LangDet	Profan	Formal
en $\rightarrow$ zh	Gemma 3 4B	1.05	0.00	0.58	0.72	1.18	0.84	0.35	0.18	0.12	0.38	0.31	1.46
	DeepSeek-V3.2	0.61	0.00	0.19	0.34	0.82	0.52	0.14	0.04	0.01	0.15	0.10	0.93
	SMART (Gemma 3 4B)	0.15	0.03	0.08	0.17	0.03	0.03	0.03	0.08	0.06	0.24	0.13	0.67
	SMART (DeepSeek-V3.2)	0.11	0.01	0.06	0.15	0.01	0.01	0.01	0.06	0.04	0.17	0.10	0.53
	SMART (Judge: GPT-5.5)	0.09	0.01	0.05	0.12	0.00	0.00	0.01	0.05	0.03	0.16	0.08	0.47

Appendix

Table 16: Fine-grained backbone and judge analysis on Subtitle Arena for translation from English into 15 target locales: Linguistic Conventions, Technical, Locale Conventions, and Audience Appropriateness. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 5 .

Dir.	Method / Setting	Terminology		Accuracy			Fluency			Overall
		NameInc	TermInc	MisTrans	UndTrans	OvrTrans	Coher	Natur	Vivid
zh $\rightarrow$ en	Gemma 3 4B	0.58	0.45	3.91	2.89	2.51	2.91	2.24	1.34	1.64
	DeepSeek-V3.2	0.36	0.26	2.74	1.86	1.50	2.11	1.39	0.78	1.07
	SMART (Gemma 3 4B)	0.25	0.13	1.25	1.09	1.02	1.44	1.04	0.69	0.62
	SMART (DeepSeek-V3.2)	0.18	0.11	1.05	0.87	0.85	1.17	0.94	0.61	0.51
	SMART (Judge: GPT-5.5)	0.15	0.09	0.90	0.72	0.67	1.04	0.84	0.57	0.44

Appendix

Table 17: Fine-grained backbone and judge analysis on Subtitle Arena for translation from 15 source locales into English: Terminology, Accuracy, and Fluency. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 5 .

Dir.	Method / Setting	Linguistic Conventions				Technical			Locale Conventions		Audience Appropriateness		Overall
		MisPunc	MisCap	Gram	Space	LineBrk	CharLim	LineLim	LocErr	LangDet	Profan	Formal
zh $\rightarrow$ en	Gemma 3 4B	1.21	0.56	0.88	0.59	1.51	1.04	0.44	0.26	0.05	0.46	0.31	1.64
	DeepSeek-V3.2	0.64	0.32	0.46	0.24	1.41	0.99	0.24	0.15	0.01	0.25	0.10	1.07
	SMART (Gemma 3 4B)	0.29	0.05	0.21	0.07	0.21	0.05	0.08	0.04	0.06	0.18	0.06	0.62
	SMART (DeepSeek-V3.2)	0.22	0.03	0.17	0.05	0.17	0.03	0.06	0.02	0.04	0.15	0.04	0.51
	SMART (Judge: GPT-5.5)	0.18	0.02	0.16	0.04	0.16	0.02	0.05	0.01	0.03	0.14	0.03	0.44

Appendix

Table 18: Fine-grained backbone and judge analysis on Subtitle Arena for translation from 15 source locales into English: Linguistic Conventions, Technical, Locale Conventions, and Audience Appropriateness. All values are penalties (lower is better). Best and second-best values within each language direction and metric are highlighted with blue-gray and warm-beige backgrounds, respectively. These results provide the full breakdown corresponding to Table 5 .

Direction	Method	Term.	Acc.	Flu.	Ling.	Tech.	Loc.	Aud.	Overall
en $\rightarrow$ zh	Online	0.28	3.52	2.95	1.42	0.12	0.85	0.08	1.87
	TransAgent	0.17	0.88	0.79	0.04	0.05	0.04	0.11	0.48
	SMART	0.16	0.83	0.71	0.03	0.00	0.05	0.10	0.44
en $\rightarrow$ de	Online	0.55	4.37	2.63	0.51	1.45	0.67	0.29	2.14
	TransAgent	0.28	1.51	1.46	0.31	1.07	0.05	0.40	0.94
	SMART	0.27	1.24	1.46	0.26	1.27	0.03	0.42	0.87

Appendix

Table 19: SubMQM results on 200 newly collected TV series released in 2025–2026. All values are penalties (lower is better). The best and second-best values for each metric within each direction are highlighted in blue-gray and warm beige, respectively.

Direction	sentences / Ep.	API Calls / Ep.	Input Tokens / Ep.	Output Tokens / Ep.	Tool Calls / Ep.	Time / Ep. (min)
en $\rightarrow$ zh	571.3	4545.9	12.83M	446.1K	2524.7	21.78
en $\rightarrow$ es	597.2	4881.1	13.78M	479.0K	3826.7	25.60
en $\rightarrow$ fr	625.4	5453.9	15.78M	535.2K	3275.3	27.34
en $\rightarrow$ de	625.4	5312.2	14.99M	521.3K	4151.4	26.97
en $\rightarrow$ it	625.4	4229.7	11.94M	415.1K	3025.7	26.75
en $\rightarrow$ ko	625.4	5394.9	15.23M	529.5K	4201.9	27.19

Appendix

Table 20: Average computational cost per episode across six translation directions. Runtime is measured under serial execution.

Figure 4: The annotation interface used in the human study. (a) Each passage is presented with its video clip, the English source lines, and the five candidate translations as anonymized columns whose order is randomized per annotation instance. (b) Annotators rank the five candidates from $5$ (best) to $1$ (worst) on Fidelity, Consistency, Language Quality, and Subtitle Quality, and then on Overall Preference; a passage can be submitted only once every panel has all five ranks assigned. The passage shown is from Nikita S01E01.

		Terminology		Accuracy			Fluency			Linguistic Conventions				Technical			Locale Conventions		Audience Appropriateness
Dir.	Method	Name	Term	Mis.	Under	Over	Coh.	Nat.	Viv.	MisP.	MisC.	Gram.	Space	Brk.	CPL	Lines	Loc.	Lang.	Prof.	Form.	Overall
en $\rightarrow$ zh	SMART	0.18	0.12	0.86	0.81	0.73	1.08	0.94	0.83	0.10	0.00	0.05	0.13	0.00	0.00	0.00	0.05	0.03	0.16	0.08	0.48
	ViDove	0.18	0.14	0.72	0.87	0.72	1.36	1.12	0.62	0.11	0.00	0.06	0.15	0.00	0.00	0.00	0.05	0.03	0.17	0.08	0.49
	Hermes	0.19	0.13	0.64	0.69	0.78	1.57	1.01	0.65	0.09	0.00	0.06	0.15	0.06	0.00	0.06	0.05	0.03	0.17	0.07	0.48
	SMART + MM	0.18	0.12	0.72	0.70	0.73	1.08	0.94	0.59	0.10	0.00	0.05	0.13	0.00	0.00	0.00	0.03	0.03	0.16	0.08	0.44
en $\rightarrow$ de	SMART	0.29	0.21	1.39	1.28	1.05	1.62	1.46	1.30	0.28	0.14	0.27	0.19	1.85	1.64	1.16	0.04	0.02	0.48	0.34	0.87

Appendix

Table 21: Fine-grained SubMQM results for multimodal expansion on six representative translation directions. We compare text-only SMART with ViDove, Hermes, and SMART augmented with video/audio context ( SMART + MM ). All values are penalties (lower is better). For clarity, only the best and second-best Overall scores within each translation direction are highlighted with blue-gray and warm-beige backgrounds, respectively.

Figure 5: A case in which visual evidence changes the translation, on the same frame for both systems. The magnifier shows a single mark on a bone, which the subtitle alone does not specify. Text-only SMART renders striation as a generic plural ( (a) , 这些条纹), while SMART + MM queries the aligned video clip, recovers that one mark on a bone is in view, and renders it as a single bone striation ( (b) , 这道骨纹). The English source is composited beneath the Chinese for reference.

Panel	Timecode	Passage	English source line	Dimension
(a)	34:05	B	Makes one of us.	Accuracy
(b)	34:51	B	We trained Nikita to be a ghost.	Terminology
(c)	34:54	B	Finding her when she doesn’t want to be found is next to impossible.	Fluency
(d)	25:14	A	The only reason why you’re alive is because she wanted you that way.	Aud. Approp.
(e)	25:10	A	Yeah, was that before or after she duct-taped you to that springy rocking horse?	Accuracy
(f)	34:45	B	Black arrow was blown.	Terminology

Appendix

Table 22: The six lines of Figures 6 and 7 , in panel order. Timecodes are episode time in Nikita S01E01; Passage indicates the ordinary-dialogue (A) or external-knowledge (B) passage of the human study; Dimension names the SubMQM dimension that the contrast bears on.

Figure 6: Rendered subtitle comparison on Nikita S01E01, English $\rightarrow$ Chinese, panels (a) and (b). Each group states the phenomenon, the timecode, and the reading the comparison turns on, so that the panels can be read without this caption. Two observations the panels do not state. In (a), Makes one of us is a sarcastic inversion of “that makes two of us,” and this is the one line in the set on which every system other than SMART agrees and is wrong. In (b), the TransAgent panel spells the protagonist’s given name with a different character than TransAgent itself uses for that name in the other two passages of the study, whereas Online, Claude Opus 4.8, and SMART use one spelling throughout; cross-passage naming consistency is what series memory (Section 3.3 ) is for, and here it is an agentic pipeline rather than a single call that fails it.

Figure 7: Rendered subtitle comparison on Nikita S01E01, English $\rightarrow$ Chinese, panels (c) and (d). Groups are annotated as in Figure 6 . These are the two panels in the set on which no system misreads the source. What separates them is how the Chinese is written: (c) is a line every baseline renders as a restated conditional that exceeds the reading-speed budget for its duration, and (d) a line every baseline renders as a literal causal clause.

Explore similar work

Towards Visually-Guided Movie Subtitle Translation for Indic Languages
May 12, 2026Tarun Chintada, Kshetrimayum Boynao Singh, Asif EkbalLow-Resource Languages Summarization
Department of Computer Science and Engineering Indian Institute of Technology Patna, India
CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles
Sep 28, 2026Mir Tafseer Nayeem, Susmoy Chakraborty, Davood RafieiCultural Awareness Long-Form Generation
University of Alberta · Independent Researcher
HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning
Jan 15, 2026Ziang Cui, Mengran Yu, Chenyu Shi +2Speech Translation
Bilibili Inc., Shanghai, China

May 12, 2026, cs.CLJ/K move · Enter open · S save

Towards Visually-Guided Movie Subtitle Translation for Indic Languages

Tarun Chintada, Kshetrimayum Boynao Singh, Asif Ekbal

Department of Computer Science and Engineering Indian Institute of Technology Patna, India

Low-Resource Languages Summarization

Open paper AskPDF arXiv

Movie subtitle translation is inherently multimodal, yet text-only systems often miss visual cues needed to convey emotion, action, and social nuance, especially for low-resource Indic languages (English to Hindi, Bengali, Telugu, Tamil and Kannada). We present a case study on five full-length films and compare two lightweight visual grounding strategies: structured attribute summaries from a 5-minute sliding window and free-text summaries of inter-subtitle visual gaps. Our analysis shows that temporal misalignment between subtitles and frames is a major obstacle in long-form video, often rendering indiscriminate visual grounding ineffective. However, oracle selective grounding, which replaces only the lowest-quality 20-30% of baseline segments with visual-enhanced outputs, consistently improves COMET over the text-only baseline while requiring far less visual processing. Among the two approaches, coarse attribute-based visual context summarization is more robust, capturing scene-level emotion and contextual subtle cues that text alone often misses

Language Direction	Method	Term.	Acc.	Flu.	Ling.	Tech.	Locale	Audience	Overall
en $\rightarrow$ zh $\|$ zh $\rightarrow$ en	Online	0.35 $\|$ 0.42	4.87 $\|$ 4.23	4.07 $\|$ 3.28	1.98 $\|$ 1.47	0.20 $\|$ 0.29	1.40 $\|$ 0.11	0.05 $\|$ 0.22	2.58 $\|$ 2.17
	Gemma 3 4B	0.49 $\|$ 0.52	2.76 $\|$ 3.10	1.97 $\|$ 2.16	0.59 $\|$ 0.81	0.79 $\|$ 1.00	0.15 $\|$ 0.16	0.35 $\|$ 0.39	1.46 $\|$ 1.64
	DeepSeek-V3.2	0.28 $\|$ 0.31	1.86 $\|$ 2.03	1.25 $\|$ 1.43	0.29 $\|$ 0.42	0.49 $\|$ 0.88	0.03 $\|$ 0.08	0.13 $\|$ 0.18	0.93 $\|$ 1.07
	Claude 4.6	0.26 $\|$ 0.27	1.63 $\|$ 1.84	1.16 $\|$ 1.28	0.26 $\|$ 0.34	0.42 $\|$ 0.83	0.03 $\|$ 0.05	0.16 $\|$ 0.15	0.84 $\|$ 0.96
	Claude 4.8	0.25 $\|$ 0.20	1.56 $\|$ 1.39	1.09 $\|$ 1.00	0.20 $\|$ 0.23	0.32 $\|$ 0.68	0.04 $\|$ 0.03	0.13 $\|$ 0.11	0.78 $\|$ 0.73
	GPT-5.5	0.20 $\|$ 0.18	1.16 $\|$ 1.20	1.13 $\|$ 0.98	0.19 $\|$ 0.19	0.22 $\|$ 0.87	0.03 $\|$ 0.03	0.14 $\|$ 0.13	0.66 $\|$ 0.67

Language Direction	Method / Setting	Term.	Acc.	Flu.	Ling.	Tech.	Locale	Audience	Overall
en $\rightarrow$ zh $\|$ zh $\rightarrow$ en	Gemma 3 4B	0.49 $\|$ 0.52	2.76 $\|$ 3.10	1.97 $\|$ 2.16	0.59 $\|$ 0.81	0.79 $\|$ 1.00	0.15 $\|$ 0.16	0.35 $\|$ 0.39	1.46 $\|$ 1.64
	DeepSeek-V3.2	0.28 $\|$ 0.31	1.86 $\|$ 2.03	1.25 $\|$ 1.43	0.29 $\|$ 0.42	0.49 $\|$ 0.88	0.03 $\|$ 0.08	0.13 $\|$ 0.18	0.93 $\|$ 1.07
	SMART (Gemma 3 4B)	0.23 $\|$ 0.19	1.14 $\|$ 1.12	1.27 $\|$ 1.06	0.11 $\|$ 0.16	0.03 $\|$ 0.11	0.07 $\|$ 0.05	0.19 $\|$ 0.12	0.67 $\|$ 0.62
	SMART (DeepSeek-V3.2)	0.18 $\|$ 0.15	0.89 $\|$ 0.92	1.04 $\|$ 0.91	0.08 $\|$ 0.12	0.01 $\|$ 0.09	0.05 $\|$ 0.03	0.14 $\|$ 0.10	0.53 $\|$ 0.51
	SMART (Judge: GPT-5.5)	0.15 $\|$ 0.12	0.80 $\|$ 0.76	0.90 $\|$ 0.82	0.07 $\|$ 0.10	0.00 $\|$ 0.08	0.04 $\|$ 0.02	0.12 $\|$ 0.09	0.47 $\|$ 0.44
	SMART (Claude 4.6)	0.15 $\|$ 0.13	0.80 $\|$ 0.80	0.95 $\|$ 0.79	0.07 $\|$ 0.10	0.00 $\|$ 0.08	0.04 $\|$ 0.02	0.12 $\|$ 0.09	0.48 $\|$ 0.44

Abstract

Explore similar work

Towards Visually-Guided Movie Subtitle Translation for Indic Languages

CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles

HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning