SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

Silvia Casola; Ryan Soh-Eun Shim; Felicia Körner; Yuchen Mao; Barbara Plank

cs.CLJan 22, 2026

SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

Authors: Silvia Casola, Ryan Soh-Eun Shim, Felicia Körner, Yuchen Mao, Barbara Plank

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Language Science and Technology, Saarland University, Germany

Abstract

An increasing body of work has leveraged multilingual language models for Natural Language Generation tasks such as summarization. A major empirical bottleneck in this area is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress. Recent studies suggest that multilingual language models often use English as an internal pivot language, and that misalignment with this pivot can lead to degraded downstream performance. Motivated by the hypothesis that this mismatch could also apply to multilingual neural metrics, we ask whether steering their activations toward an English pivot can improve correlation with human judgments. We experiment with encoder- and decoder-based metrics and find that test-time intervention methods are effective across the board, increasing metric effectiveness for diverse languages.

Figures & tables

Figure 1: Illustration of our SteerEval method. We first use parallel data to learn cross-lingual vectors of mappings from Source language X $\rightarrow$ English or a single multilingual average vector. We then apply the steering to neural metrics. Finally, we show that such steering results in better correlation with human judgments.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$.022_{\pm.132}$	$.213_{\pm.141}$	$.270_{\pm.062}$	$.202_{\pm.108}$	$\underline{.139}_{\pm.092}$
Llama-3.2-3B	+ vector steer (EN-prompt)	$.022_{\pm.127}$	$.215_{\pm.142}$ $\uparrow$	$.276_{\pm.062}$ $\uparrow$	$\underline{.211}_{\pm.100}$ $\uparrow$	$.138_{\pm.095}$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$\underline{.024}_{\pm.127}$ $\uparrow$	$\underline{.221}_{\pm.149}$ $\uparrow$	$\underline{.281}_{\pm.062}$ $\uparrow$	$.210_{\pm.104}$ $\uparrow$	$\underline{.139}_{\pm.097}$
aya-expanse-8b	Baseline (EN-prompt)	$.017_{\pm.143}$	$.150_{\pm.088}$	$.240_{\pm.067}$	$.190_{\pm.069}$	$.181_{\pm.063}$
aya-expanse-8b	+ vector steer (EN-prompt)	$.018_{\pm.145}$ $\uparrow$	$.155_{\pm.088}$ $\uparrow$	$.247_{\pm.067}$ $\uparrow$	$.198_{\pm.070}$ $\uparrow$	$.185_{\pm.063}$ $\uparrow$

Table 1: SEAHORSE Pearson

r

averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (resp. degradation) relative to the baseline.

	de	es	ru	tr	vi
COMET
Baseline	$.248_{\pm.087}$	$.266_{\pm.120}$	$.358_{\pm.130}$	$.380_{\pm.139}$	$.290_{\pm.116}$
+ vector steer	$\mathbf{.279}_{\pm.081}$ $\uparrow$	$\mathbf{.318}_{\pm.184}$ $\uparrow$	$.351_{\pm.155}$ $\downarrow$	$.360_{\pm.144}$ $\downarrow$	$.295_{\pm.136}$ $\uparrow$
+ avgvec zero-shot	$.275_{\pm.084}$ $\uparrow$	$.303_{\pm.146}$ $\uparrow$	$\mathbf{.364}_{\pm.152}$ $\uparrow$	$\mathbf{.392}_{\pm.165}$ $\uparrow$	$\mathbf{.297}_{\pm.134}$ $\uparrow$

Table 2: SEAHORSE Pearson

r

over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language for COMET ( wmt22-comet-da ). Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the highest value per language. Blue /

\uparrow

( red /

\downarrow

) indicates improvement (resp. degradation) subscripts relative to the baseline.

Panel of Figure 2 — Figure 2: Projection of the instances from FLORES into the Principal Components (left) and cosine similarity across steering vectors (right) for Llama-3.2-3B.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$.003$	$.319$	$.313$	$.099$	$.248$
	+ vector steer (EN-prompt)	$\underline{.025}$ $\uparrow$	$.320$ $\uparrow$	$.315$ $\uparrow$	$\underline{.158}$ $\uparrow$	$.244$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$.023$ $\uparrow$	$\underline{.333}$ $\uparrow$	$\underline{.331}$ $\uparrow$	$.140$ $\uparrow$	$\underline{.258}$ $\uparrow$
aya-expanse-8b	Baseline (EN-prompt)	$\underline{.029}$	$.249$	$.353$	$.114$	$.305$
	+ vector steer (EN-prompt)	$.027$ $\downarrow$	$.255$ $\uparrow$	$.361$ $\uparrow$	$\underline{.118}$ $\uparrow$	$.308$ $\uparrow$

Appendix

Table 3: SEAHORSE Pearson

r

for dimension Comprehensible per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split; avgvec zero-shot uses a fixed vector averaged across all source languages. Bold indicates the highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$-.147$	$.068$	$.178$	$\underline{.095}$	$.079$
	+ vector steer (EN-prompt)	$-.145$ $\uparrow$	$.067$ $\downarrow$	$.185$ $\uparrow$	$.094$ $\downarrow$	$\underline{.079}$ $\uparrow$
	+ avgvec zero-shot (EN-prompt)	$\underline{-.139}$ $\uparrow$	$\underline{.075}$ $\uparrow$	$\underline{.191}$ $\uparrow$	$\underline{.095}$ $\downarrow$	$.075$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$-.207$	$.033$	$.234$	$.168$	$.193$
	+ vector steer (EN-prompt)	$-.206$ $\uparrow$	$.038$ $\uparrow$	$.242$ $\uparrow$	$.176$ $\uparrow$	$.196$ $\uparrow$

Appendix

Table 4: SEAHORSE Pearson

r

for dimension Non-repetitive per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$-.073$	$\underline{-.023}$	$.190$	$.090$	$.124$
	+ vector steer (EN-prompt)	$-.074$ $\downarrow$	$\underline{-.023}$ $\uparrow$	$.197$ $\uparrow$	$.090$ $\downarrow$	$\underline{.125}$ $\uparrow$
	+ avgvec zero-shot (EN-prompt)	$\underline{-.070}$ $\uparrow$	$-.033$ $\downarrow$	$\mathbf{\underline{.198}}$ $\uparrow$	$\underline{.092}$ $\uparrow$	$.121$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$-.101$	$.065$	$.140$	$.121$	$.171$
	+ vector steer (EN-prompt)	$-.101$ $\uparrow$	$\mathbf{\underline{.068}}$ $\uparrow$	$.148$ $\uparrow$	$.126$ $\uparrow$	$.174$ $\uparrow$

Appendix

Table 5: SEAHORSE Pearson

r

for dimension Grammatical per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.279}$	$.368$	$.334$	$.336$	$.278$
	+ vector steer (EN-prompt)	$.268$ $\downarrow$	$.371$ $\uparrow$	$\mathbf{\underline{.342}}$ $\uparrow$	$.334$ $\downarrow$	$\underline{.284}$ $\uparrow$
	+ avgvec zero-shot (EN-prompt)	$.272$ $\downarrow$	$\mathbf{\underline{.388}}$ $\uparrow$	$.341$ $\uparrow$	$\mathbf{\underline{.341}}$ $\uparrow$	$.280$ $\uparrow$
aya-expanse-8b	Baseline (EN-prompt)	$.249$	$.101$	$.190$	$.197$	$.182$
	+ vector steer (EN-prompt)	$.254$ $\uparrow$	$.107$ $\uparrow$	$.196$ $\uparrow$	$\underline{.208}$ $\uparrow$	$.186$ $\uparrow$

Appendix

Table 6: SEAHORSE Pearson

r

for dimension Attributable per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.023}$	$.267$	$.303$	$\underline{.290}$	$\underline{.029}$
	+ vector steer (EN-prompt)	$.021$ $\downarrow$	$.271$ $\uparrow$	$\underline{.308}$ $\uparrow$	$.285$ $\downarrow$	$.021$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$.020$ $\downarrow$	$\underline{.272}$ $\uparrow$	$\underline{.308}$ $\uparrow$	$.288$ $\downarrow$	$.022$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$.073$	$.258$	$.284$	$.317$	$.136$
	+ vector steer (EN-prompt)	$.074$ $\uparrow$	$.261$ $\uparrow$	$\underline{.289}$ $\uparrow$	$.324$ $\uparrow$	$.138$ $\uparrow$

Appendix

Table 7: SEAHORSE Pearson

r

for dimension Main ideas per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.045}$	$.279$	$.303$	$.303$	$\underline{.079}$
	+ vector steer (EN-prompt)	$.035$ $\downarrow$	$.284$ $\uparrow$	$.312$ $\uparrow$	$.301$ $\downarrow$	$.075$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$.037$ $\downarrow$	$\mathbf{\underline{.290}}$ $\uparrow$	$\underline{.314}$ $\uparrow$	$\mathbf{\underline{.306}}$ $\uparrow$	$.075$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$.058$	$.195$	$.237$	$.223$	$.102$
	+ vector steer (EN-prompt)	$\underline{.062}$ $\uparrow$	$.199$ $\uparrow$	$.244$ $\uparrow$	$.235$ $\uparrow$	$.106$ $\uparrow$

Appendix

Table 8: SEAHORSE Pearson

r

for dimension Concise per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de		es		ru		tr		vi
		layer	$\sigma$	layer	$\sigma$	layer	$\sigma$	layer	$\sigma$	layer	$\sigma$
Llama-3.2-3B	EN-prompt	27	$-2.0$	27	$+2.0$	27	$-1.0$	27	$-5.0$	22	$-2.0$
Llama-3.2-3B	src-prompt	27	$-2.0$	27	$+1.0$	27	$-1.0$	27	$-3.0$	27	$+1.0$
aya-expanse-8b	EN-prompt	27	$-5.0$	31	$-3.0$	31	$-2.0$	31	$-3.0$	31	$-2.0$
aya-expanse-8b	src-prompt	31	$-3.0$	31	$-2.0$	31	$-1.0$	31	$-2.0$	31	$-1.0$

Appendix

Table 9: Best intervention layer and steering

\sigma

for LLM-based SEAHORSE evaluation. Layer and

\sigma

are selected jointly on the validation split by mean Pearson

r

across six quality dimensions. Avgvec rows use a fixed

\sigma

and the layer that maximised cross-lingual vector similarity.

COMET (wmt22-comet-da)
	de	es	ru	tr	vi
vector steer	$-5.0$	$-5.0$	$-2.0$	$+5.0$	$-2.0$

Appendix

Table 10: Best steering

\sigma

for COMET ( wmt22-comet-da ) SEAHORSE evaluation. COMET steers the pooling-space embedding directly (no per-layer selection);

\sigma

is chosen on the validation split by mean Pearson

r

across six quality dimensions.

Figure 3: GPTScore prompt templates per quality dimension and language. [article] is replaced by the source article; the second line (after the newline) is the continuation cue used for scoring.

Figure 4: Direct Prompting (G-eval) question per quality dimension and language.

Figure 5: Fixed sentence-level labels used in Direct Prompting per language.

System	DE	EN	ES	RU	TR	VI
T5-Base	0	236	0	0	0	0
T5-Base (250s)	0	241	0	0	0	0
T5-XXL	0	247	0	0	0	0
mT5-Small	231	239	234	222	218	234
mT5-Small (250s)	226	243	239	218	209	229
mT5-XXL	228	239	242	208	208	238

Appendix

Table 11: Number of outputs samples, annotated with human judgments for the Seahorse validation set.

System	DE	EN	ES	RU	TR	VI
T5-Base	0	484	0	0	0	0
T5-Base (250s)	0	482	0	0	0	0
T5-XXL	0	476	0	0	0	0
mT5-Small	451	482	474	449	495	497
mT5-Small (250s)	459	484	477	427	488	490
mT5-XXL	443	481	479	450	485	492

Appendix

Table 12: Number of outputs samples, annotated with human judgments for the Seahorse test set.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (src-prompt)	$\underline{.028}_{\pm.125}$	$.213_{\pm.143}$	$.273_{\pm.065}$	$.204_{\pm.103}$	$\underline{.144}_{\pm.093}$
Llama-3.2-3B	+ vector steer (src-prompt)	$.026_{\pm.127}$ $\downarrow$	$\underline{.215}_{\pm.141}$ $\uparrow$	$\mathbf{\underline{.278}}_{\pm.064}$ $\uparrow$	$\mathbf{\underline{.212}}_{\pm.100}$ $\uparrow$	$.143_{\pm.090}$ $\downarrow$
aya-expanse-8b	Baseline (src-prompt)	$.004_{\pm.145}$	$.129_{\pm.079}$	$.195_{\pm.065}$	$.154_{\pm.069}$	$.158_{\pm.068}$
aya-expanse-8b	+ vector steer (src-prompt)	$\underline{.006}_{\pm.144}$ $\uparrow$	$\underline{.137}_{\pm.081}$ $\uparrow$	$\underline{.200}_{\pm.065}$ $\uparrow$	$\underline{.160}_{\pm.070}$ $\uparrow$	$\underline{.161}_{\pm.067}$ $\uparrow$
Direct Prompting

Appendix

Table 13: SEAHORSE Pearson

r

averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using source-language prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.483}_{\pm.112}$	$.631_{\pm.080}$	$.698_{\pm.024}$	$\underline{.628}_{\pm.069}$	$\underline{.604}_{\pm.051}$
Llama-3.2-3B	+ vector steer (EN-prompt)	$.483_{\pm.111}$ $\downarrow$	$\underline{.632}_{\pm.082}$ $\uparrow$	$\mathbf{\underline{.702}}_{\pm.024}$ $\uparrow$	$.626_{\pm.069}$ $\downarrow$	$.603_{\pm.052}$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$.457_{\pm.135}$	$.597_{\pm.046}$	$.679_{\pm.055}$	$.625_{\pm.060}$	$.641_{\pm.063}$
aya-expanse-8b	+ vector steer (EN-prompt)	$\underline{.459}_{\pm.136}$ $\uparrow$	$\underline{.600}_{\pm.046}$ $\uparrow$	$\underline{.684}_{\pm.056}$ $\uparrow$	$\underline{.631}_{\pm.059}$ $\uparrow$	$\underline{.643}_{\pm.064}$ $\uparrow$
Direct Prompting

Appendix

Table 14: SEAHORSE AUC-ROC averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (src-prompt)	$\underline{.484}_{\pm.108}$	$.631_{\pm.083}$	$.697_{\pm.026}$	$\mathbf{\underline{.631}}_{\pm.065}$	$\underline{.607}_{\pm.052}$
Llama-3.2-3B	+ vector steer (src-prompt)	$.482_{\pm.112}$ $\downarrow$	$\underline{.633}_{\pm.082}$ $\uparrow$	$\mathbf{\underline{.701}}_{\pm.027}$ $\uparrow$	$\mathbf{\underline{.631}}_{\pm.066}$ $\downarrow$	$.607_{\pm.050}$ $\downarrow$
aya-expanse-8b	Baseline (src-prompt)	$.451_{\pm.138}$	$.582_{\pm.041}$	$.651_{\pm.059}$	$.600_{\pm.060}$	$.627_{\pm.061}$
aya-expanse-8b	+ vector steer (src-prompt)	$\underline{.453}_{\pm.137}$ $\uparrow$	$\underline{.588}_{\pm.043}$ $\uparrow$	$\underline{.655}_{\pm.060}$ $\uparrow$	$\underline{.604}_{\pm.059}$ $\uparrow$	$\underline{.629}_{\pm.062}$ $\uparrow$
Direct Prompting

Appendix

Table 15: SEAHORSE AUC-ROC averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using source-language prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

		de	es	ru	tr	vi
GPTScore
Llama-3.1-8B-Instruct	Baseline (EN-prompt)	$.480_{\pm.120}$	$.643_{\pm.077}$	$.711_{\pm.028}$	$.659_{\pm.059}$	$\underline{.631}_{\pm.051}$
Llama-3.1-8B-Instruct	+ avgvec zero-shot (EN-prompt)	$\mathbf{\underline{.487}}_{\pm.113}$ $\uparrow$	$\mathbf{\underline{.649}}_{\pm.084}$ $\uparrow$	$\mathbf{\underline{.717}}_{\pm.029}$ $\uparrow$	$\mathbf{\underline{.660}}_{\pm.058}$ $\uparrow$	$.629_{\pm.051}$ $\downarrow$
aya-expanse-32b	Baseline (EN-prompt)	$.448_{\pm.131}$	$.603_{\pm.039}$	$.680_{\pm.048}$	$.626_{\pm.061}$	$.642_{\pm.055}$
aya-expanse-32b	+ avgvec zero-shot (EN-prompt)	$\underline{.451}_{\pm.131}$ $\uparrow$	$\underline{.607}_{\pm.040}$ $\uparrow$	$\underline{.683}_{\pm.049}$ $\uparrow$	$\underline{.631}_{\pm.060}$ $\uparrow$	$\mathbf{\underline{.645}}_{\pm.056}$ $\uparrow$

Appendix

Table 16: SEAHORSE AUC-ROC averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); avgvec zero-shot uses a fixed steering vector averaged across all source languages with a pre-set

\sigma

. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

Summarization Evaluation

Summarization evaluation measures whether summaries preserve important information, remain faithful to their sources, and serve readers’ needs.

See all 44 papers

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization
May 31, 2026Sangwon Ryu, Yihong Liu, Mingyang Wang +4Multilingual Language Model Evaluation Text Summarization
GSAI, POSTECH · LMU Munich · MCML +2
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM Evaluation LLM-as-a-Judge
University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands
Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference
Aug 5, 2026Hongsheng Wang, Philipp KoehnMultilingual Language Models Language Model Steering
Johns Hopkins University, Baltimore / USA

May 31, 2026J/K move · Enter open · S save

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

Sangwon Ryu, Yihong Liu, Mingyang Wang, Yunsu Kim+3

GSAI, POSTECH · LMU Munich · MCML+2

Multilingual Language Model Evaluation Text Summarization Cross-Lingual Transfer

Open paper AskPDF arXiv

Multi-target cross-lingual text summarization (MTXLS), which summarizes a source document into multiple target languages, is increasingly important as users consume content in diverse languages, but remains underexplored. To address this gap, we introduce multi-target cross-lingual element-aware (MEA), a new MTXLS benchmark covering 24 target languages. We benchmark end-to-end and pipeline approaches across various LLMs and show that MTXLS performance still substantially lags behind English monolingual summarization. To better understand MTXLS in LLMs, we propose a layer-wise analysis framework for investigating how LLMs internally perform MTXLS. Our analyses suggest that translation and summarization behaviors emerge jointly within later layers rather than as distinctly decomposed stages. Most task-relevant processing occurs within these layers, and errors also tend to arise at similar depths. Motivated by these findings, we introduce an inference-time activation steering method that leverages hidden representations from English summarization to guide MTXLS generation. Experiments show that our method consistently improves MTXLS quality across target languages.

Abstract

Summarization Evaluation

Explore similar work

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference