SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

Silvia Casola; Ryan Soh-Eun Shim; Felicia Körner; Yuchen Mao; Barbara Plank

cs.CLJan 22, 2026

SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

Authors: Silvia Casola, Ryan Soh-Eun Shim, Felicia Körner, Yuchen Mao, Barbara Plank

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Language Science and Technology, Saarland University, Germany

Abstract

An increasing body of work has leveraged multilingual language models for Natural Language Generation tasks such as summarization. A major empirical bottleneck in this area is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress. Recent studies suggest that multilingual language models often use English as an internal pivot language, and that misalignment with this pivot can lead to degraded downstream performance. Motivated by the hypothesis that this mismatch could also apply to multilingual neural metrics, we ask whether steering their activations toward an English pivot can improve correlation with human judgments. We experiment with encoder- and decoder-based metrics and find that test-time intervention methods are effective across the board, increasing metric effectiveness for diverse languages.

Figures & tables

Figure 1: Illustration of our SteerEval method. We first use parallel data to learn cross-lingual vectors of mappings from Source language X $\rightarrow$ English or a single multilingual average vector. We then apply the steering to neural metrics. Finally, we show that such steering results in better correlation with human judgments.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$.022_{\pm.132}$	$.213_{\pm.141}$	$.270_{\pm.062}$	$.202_{\pm.108}$	$\underline{.139}_{\pm.092}$
Llama-3.2-3B	+ vector steer (EN-prompt)	$.022_{\pm.127}$	$.215_{\pm.142}$ $\uparrow$	$.276_{\pm.062}$ $\uparrow$	$\underline{.211}_{\pm.100}$ $\uparrow$	$.138_{\pm.095}$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$\underline{.024}_{\pm.127}$ $\uparrow$	$\underline{.221}_{\pm.149}$ $\uparrow$	$\underline{.281}_{\pm.062}$ $\uparrow$	$.210_{\pm.104}$ $\uparrow$	$\underline{.139}_{\pm.097}$
aya-expanse-8b	Baseline (EN-prompt)	$.017_{\pm.143}$	$.150_{\pm.088}$	$.240_{\pm.067}$	$.190_{\pm.069}$	$.181_{\pm.063}$
aya-expanse-8b	+ vector steer (EN-prompt)	$.018_{\pm.145}$ $\uparrow$	$.155_{\pm.088}$ $\uparrow$	$.247_{\pm.067}$ $\uparrow$	$.198_{\pm.070}$ $\uparrow$	$.185_{\pm.063}$ $\uparrow$

Table 1: SEAHORSE Pearson

r

averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (resp. degradation) relative to the baseline.

	de	es	ru	tr	vi
COMET
Baseline	$.248_{\pm.087}$	$.266_{\pm.120}$	$.358_{\pm.130}$	$.380_{\pm.139}$	$.290_{\pm.116}$
+ vector steer	$\mathbf{.279}_{\pm.081}$ $\uparrow$	$\mathbf{.318}_{\pm.184}$ $\uparrow$	$.351_{\pm.155}$ $\downarrow$	$.360_{\pm.144}$ $\downarrow$	$.295_{\pm.136}$ $\uparrow$
+ avgvec zero-shot	$.275_{\pm.084}$ $\uparrow$	$.303_{\pm.146}$ $\uparrow$	$\mathbf{.364}_{\pm.152}$ $\uparrow$	$\mathbf{.392}_{\pm.165}$ $\uparrow$	$\mathbf{.297}_{\pm.134}$ $\uparrow$

Table 2: SEAHORSE Pearson

r

over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language for COMET ( wmt22-comet-da ). Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the highest value per language. Blue /

\uparrow

( red /

\downarrow

) indicates improvement (resp. degradation) subscripts relative to the baseline.

Panel of Figure 2 — Figure 2: Projection of the instances from FLORES into the Principal Components (left) and cosine similarity across steering vectors (right) for Llama-3.2-3B.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$.003$	$.319$	$.313$	$.099$	$.248$
	+ vector steer (EN-prompt)	$\underline{.025}$ $\uparrow$	$.320$ $\uparrow$	$.315$ $\uparrow$	$\underline{.158}$ $\uparrow$	$.244$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$.023$ $\uparrow$	$\underline{.333}$ $\uparrow$	$\underline{.331}$ $\uparrow$	$.140$ $\uparrow$	$\underline{.258}$ $\uparrow$
aya-expanse-8b	Baseline (EN-prompt)	$\underline{.029}$	$.249$	$.353$	$.114$	$.305$
	+ vector steer (EN-prompt)	$.027$ $\downarrow$	$.255$ $\uparrow$	$.361$ $\uparrow$	$\underline{.118}$ $\uparrow$	$.308$ $\uparrow$

Appendix

Table 3: SEAHORSE Pearson

r

for dimension Comprehensible per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split; avgvec zero-shot uses a fixed vector averaged across all source languages. Bold indicates the highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$-.147$	$.068$	$.178$	$\underline{.095}$	$.079$
	+ vector steer (EN-prompt)	$-.145$ $\uparrow$	$.067$ $\downarrow$	$.185$ $\uparrow$	$.094$ $\downarrow$	$\underline{.079}$ $\uparrow$
	+ avgvec zero-shot (EN-prompt)	$\underline{-.139}$ $\uparrow$	$\underline{.075}$ $\uparrow$	$\underline{.191}$ $\uparrow$	$\underline{.095}$ $\downarrow$	$.075$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$-.207$	$.033$	$.234$	$.168$	$.193$
	+ vector steer (EN-prompt)	$-.206$ $\uparrow$	$.038$ $\uparrow$	$.242$ $\uparrow$	$.176$ $\uparrow$	$.196$ $\uparrow$

Appendix

Table 4: SEAHORSE Pearson

r

for dimension Non-repetitive per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$-.073$	$\underline{-.023}$	$.190$	$.090$	$.124$
	+ vector steer (EN-prompt)	$-.074$ $\downarrow$	$\underline{-.023}$ $\uparrow$	$.197$ $\uparrow$	$.090$ $\downarrow$	$\underline{.125}$ $\uparrow$
	+ avgvec zero-shot (EN-prompt)	$\underline{-.070}$ $\uparrow$	$-.033$ $\downarrow$	$\mathbf{\underline{.198}}$ $\uparrow$	$\underline{.092}$ $\uparrow$	$.121$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$-.101$	$.065$	$.140$	$.121$	$.171$
	+ vector steer (EN-prompt)	$-.101$ $\uparrow$	$\mathbf{\underline{.068}}$ $\uparrow$	$.148$ $\uparrow$	$.126$ $\uparrow$	$.174$ $\uparrow$

Appendix

Table 5: SEAHORSE Pearson

r

for dimension Grammatical per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.279}$	$.368$	$.334$	$.336$	$.278$
	+ vector steer (EN-prompt)	$.268$ $\downarrow$	$.371$ $\uparrow$	$\mathbf{\underline{.342}}$ $\uparrow$	$.334$ $\downarrow$	$\underline{.284}$ $\uparrow$
	+ avgvec zero-shot (EN-prompt)	$.272$ $\downarrow$	$\mathbf{\underline{.388}}$ $\uparrow$	$.341$ $\uparrow$	$\mathbf{\underline{.341}}$ $\uparrow$	$.280$ $\uparrow$
aya-expanse-8b	Baseline (EN-prompt)	$.249$	$.101$	$.190$	$.197$	$.182$
	+ vector steer (EN-prompt)	$.254$ $\uparrow$	$.107$ $\uparrow$	$.196$ $\uparrow$	$\underline{.208}$ $\uparrow$	$.186$ $\uparrow$

Appendix

Table 6: SEAHORSE Pearson

r

for dimension Attributable per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.023}$	$.267$	$.303$	$\underline{.290}$	$\underline{.029}$
	+ vector steer (EN-prompt)	$.021$ $\downarrow$	$.271$ $\uparrow$	$\underline{.308}$ $\uparrow$	$.285$ $\downarrow$	$.021$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$.020$ $\downarrow$	$\underline{.272}$ $\uparrow$	$\underline{.308}$ $\uparrow$	$.288$ $\downarrow$	$.022$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$.073$	$.258$	$.284$	$.317$	$.136$
	+ vector steer (EN-prompt)	$.074$ $\uparrow$	$.261$ $\uparrow$	$\underline{.289}$ $\uparrow$	$.324$ $\uparrow$	$.138$ $\uparrow$

Appendix

Table 7: SEAHORSE Pearson

r

for dimension Main ideas per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.045}$	$.279$	$.303$	$.303$	$\underline{.079}$
	+ vector steer (EN-prompt)	$.035$ $\downarrow$	$.284$ $\uparrow$	$.312$ $\uparrow$	$.301$ $\downarrow$	$.075$ $\downarrow$
	+ avgvec zero-shot (EN-prompt)	$.037$ $\downarrow$	$\mathbf{\underline{.290}}$ $\uparrow$	$\underline{.314}$ $\uparrow$	$\mathbf{\underline{.306}}$ $\uparrow$	$.075$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$.058$	$.195$	$.237$	$.223$	$.102$
	+ vector steer (EN-prompt)	$\underline{.062}$ $\uparrow$	$.199$ $\uparrow$	$.244$ $\uparrow$	$.235$ $\uparrow$	$.106$ $\uparrow$

Appendix

Table 8: SEAHORSE Pearson

r

for dimension Concise per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de		es		ru		tr		vi
		layer	$\sigma$	layer	$\sigma$	layer	$\sigma$	layer	$\sigma$	layer	$\sigma$
Llama-3.2-3B	EN-prompt	27	$-2.0$	27	$+2.0$	27	$-1.0$	27	$-5.0$	22	$-2.0$
Llama-3.2-3B	src-prompt	27	$-2.0$	27	$+1.0$	27	$-1.0$	27	$-3.0$	27	$+1.0$
aya-expanse-8b	EN-prompt	27	$-5.0$	31	$-3.0$	31	$-2.0$	31	$-3.0$	31	$-2.0$
aya-expanse-8b	src-prompt	31	$-3.0$	31	$-2.0$	31	$-1.0$	31	$-2.0$	31	$-1.0$

Appendix

Table 9: Best intervention layer and steering

\sigma

for LLM-based SEAHORSE evaluation. Layer and

\sigma

are selected jointly on the validation split by mean Pearson

r

across six quality dimensions. Avgvec rows use a fixed

\sigma

and the layer that maximised cross-lingual vector similarity.

COMET (wmt22-comet-da)
	de	es	ru	tr	vi
vector steer	$-5.0$	$-5.0$	$-2.0$	$+5.0$	$-2.0$

Appendix

Table 10: Best steering

\sigma

for COMET ( wmt22-comet-da ) SEAHORSE evaluation. COMET steers the pooling-space embedding directly (no per-layer selection);

\sigma

is chosen on the validation split by mean Pearson

r

across six quality dimensions.

Figure 3: GPTScore prompt templates per quality dimension and language. [article] is replaced by the source article; the second line (after the newline) is the continuation cue used for scoring.

Figure 4: Direct Prompting (G-eval) question per quality dimension and language.

Figure 5: Fixed sentence-level labels used in Direct Prompting per language.

System	DE	EN	ES	RU	TR	VI
T5-Base	0	236	0	0	0	0
T5-Base (250s)	0	241	0	0	0	0
T5-XXL	0	247	0	0	0	0
mT5-Small	231	239	234	222	218	234
mT5-Small (250s)	226	243	239	218	209	229
mT5-XXL	228	239	242	208	208	238

Appendix

Table 11: Number of outputs samples, annotated with human judgments for the Seahorse validation set.

System	DE	EN	ES	RU	TR	VI
T5-Base	0	484	0	0	0	0
T5-Base (250s)	0	482	0	0	0	0
T5-XXL	0	476	0	0	0	0
mT5-Small	451	482	474	449	495	497
mT5-Small (250s)	459	484	477	427	488	490
mT5-XXL	443	481	479	450	485	492

Appendix

Table 12: Number of outputs samples, annotated with human judgments for the Seahorse test set.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (src-prompt)	$\underline{.028}_{\pm.125}$	$.213_{\pm.143}$	$.273_{\pm.065}$	$.204_{\pm.103}$	$\underline{.144}_{\pm.093}$
Llama-3.2-3B	+ vector steer (src-prompt)	$.026_{\pm.127}$ $\downarrow$	$\underline{.215}_{\pm.141}$ $\uparrow$	$\mathbf{\underline{.278}}_{\pm.064}$ $\uparrow$	$\mathbf{\underline{.212}}_{\pm.100}$ $\uparrow$	$.143_{\pm.090}$ $\downarrow$
aya-expanse-8b	Baseline (src-prompt)	$.004_{\pm.145}$	$.129_{\pm.079}$	$.195_{\pm.065}$	$.154_{\pm.069}$	$.158_{\pm.068}$
aya-expanse-8b	+ vector steer (src-prompt)	$\underline{.006}_{\pm.144}$ $\uparrow$	$\underline{.137}_{\pm.081}$ $\uparrow$	$\underline{.200}_{\pm.065}$ $\uparrow$	$\underline{.160}_{\pm.070}$ $\uparrow$	$\underline{.161}_{\pm.067}$ $\uparrow$
Direct Prompting

Appendix

Table 13: SEAHORSE Pearson

r

averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using source-language prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (EN-prompt)	$\underline{.483}_{\pm.112}$	$.631_{\pm.080}$	$.698_{\pm.024}$	$\underline{.628}_{\pm.069}$	$\underline{.604}_{\pm.051}$
Llama-3.2-3B	+ vector steer (EN-prompt)	$.483_{\pm.111}$ $\downarrow$	$\underline{.632}_{\pm.082}$ $\uparrow$	$\mathbf{\underline{.702}}_{\pm.024}$ $\uparrow$	$.626_{\pm.069}$ $\downarrow$	$.603_{\pm.052}$ $\downarrow$
aya-expanse-8b	Baseline (EN-prompt)	$.457_{\pm.135}$	$.597_{\pm.046}$	$.679_{\pm.055}$	$.625_{\pm.060}$	$.641_{\pm.063}$
aya-expanse-8b	+ vector steer (EN-prompt)	$\underline{.459}_{\pm.136}$ $\uparrow$	$\underline{.600}_{\pm.046}$ $\uparrow$	$\underline{.684}_{\pm.056}$ $\uparrow$	$\underline{.631}_{\pm.059}$ $\uparrow$	$\underline{.643}_{\pm.064}$ $\uparrow$
Direct Prompting

Appendix

Table 14: SEAHORSE AUC-ROC averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

GPTScore
		de	es	ru	tr	vi
Llama-3.2-3B	Baseline (src-prompt)	$\underline{.484}_{\pm.108}$	$.631_{\pm.083}$	$.697_{\pm.026}$	$\mathbf{\underline{.631}}_{\pm.065}$	$\underline{.607}_{\pm.052}$
Llama-3.2-3B	+ vector steer (src-prompt)	$.482_{\pm.112}$ $\downarrow$	$\underline{.633}_{\pm.082}$ $\uparrow$	$\mathbf{\underline{.701}}_{\pm.027}$ $\uparrow$	$\mathbf{\underline{.631}}_{\pm.066}$ $\downarrow$	$.607_{\pm.050}$ $\downarrow$
aya-expanse-8b	Baseline (src-prompt)	$.451_{\pm.138}$	$.582_{\pm.041}$	$.651_{\pm.059}$	$.600_{\pm.060}$	$.627_{\pm.061}$
aya-expanse-8b	+ vector steer (src-prompt)	$\underline{.453}_{\pm.137}$ $\uparrow$	$\underline{.588}_{\pm.043}$ $\uparrow$	$\underline{.655}_{\pm.060}$ $\uparrow$	$\underline{.604}_{\pm.059}$ $\uparrow$	$\underline{.629}_{\pm.062}$ $\uparrow$
Direct Prompting

Appendix

Table 15: SEAHORSE AUC-ROC averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using source-language prompts. Baseline uses

\sigma{=}0

(no steering); steered uses the best

\sigma

selected on the validation split. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

		de	es	ru	tr	vi
GPTScore
Llama-3.1-8B-Instruct	Baseline (EN-prompt)	$.480_{\pm.120}$	$.643_{\pm.077}$	$.711_{\pm.028}$	$.659_{\pm.059}$	$\underline{.631}_{\pm.051}$
Llama-3.1-8B-Instruct	+ avgvec zero-shot (EN-prompt)	$\mathbf{\underline{.487}}_{\pm.113}$ $\uparrow$	$\mathbf{\underline{.649}}_{\pm.084}$ $\uparrow$	$\mathbf{\underline{.717}}_{\pm.029}$ $\uparrow$	$\mathbf{\underline{.660}}_{\pm.058}$ $\uparrow$	$.629_{\pm.051}$ $\downarrow$
aya-expanse-32b	Baseline (EN-prompt)	$.448_{\pm.131}$	$.603_{\pm.039}$	$.680_{\pm.048}$	$.626_{\pm.061}$	$.642_{\pm.055}$
aya-expanse-32b	+ avgvec zero-shot (EN-prompt)	$\underline{.451}_{\pm.131}$ $\uparrow$	$\underline{.607}_{\pm.040}$ $\uparrow$	$\underline{.683}_{\pm.049}$ $\uparrow$	$\underline{.631}_{\pm.060}$ $\uparrow$	$\mathbf{\underline{.645}}_{\pm.056}$ $\uparrow$

Appendix

Table 16: SEAHORSE AUC-ROC averaged over 6 quality dimensions (Comprehensible, Non-repetitive, Grammatical, Attributable, Main ideas, Concise) with standard deviation as subscripts (

\text{mean}_{\pm\text{std}}

), per language, using English prompts. Baseline uses

\sigma{=}0

(no steering); avgvec zero-shot uses a fixed steering vector averaged across all source languages with a pre-set

\sigma

. Bold indicates the single highest value per language across the entire table; underline indicates the highest value per language within each model. Blue /

\uparrow

( red /

\downarrow

) on steered rows indicates improvement (degradation) relative to the baseline.

Summarization Evaluation

Summarization evaluation measures whether summaries preserve important information, remain faithful to their sources, and serve readers’ needs.

See all 44 papers

May 31, 2026

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

Multi-target cross-lingual text summarization (MTXLS), which summarizes a source document into multiple target languages, is increasingly important as users consume content in diverse languages, but remains underexplored. To address this gap, we introduce multi-target cross-lingual element-aware (MEA), a new MTXLS benchmark covering 24 target languages. We benchmark end-to-end and pipeline approaches across various LLMs and show that MTXLS performance still substantially lags behind English monolingual summarization. To better understand MTXLS in LLMs, we propose a layer-wise analysis framework for investigating how LLMs internally perform MTXLS. Our analyses suggest that translation and summarization behaviors emerge jointly within later layers rather than as distinctly decomposed stages. Most task-relevant processing occurs within these layers, and errors also tend to arise at similar depths. Motivated by these findings, we introduce an inference-time activation steering method that leverages hidden representations from English summarization to guide MTXLS generation. Experiments show that our method consistently improves MTXLS quality across target languages.

Sangwon Ryu, Yihong Liu, Mingyang Wang +4

GSAI, POSTECH · LMU Munich · MCML +2

Multilingual Language Model Evaluation Text Summarization Cross-Lingual Transfer

Ask PDF

Oct 22, 2026

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro

University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

LLM Evaluation LLM-as-a-Judge Multilingual Language Model Evaluation

Ask PDF

Aug 5, 2026

Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show macro-average accuracy improvements of 11.28 percentage points on XCOPA, 4.11 points on XNLI, and 7.84 points on PAWS-X. Additional evaluation with Qwen2.5-7B-Instruct yields a macro-average improvement of 3.33 percentage points on XCOPA.

Hongsheng Wang, Philipp Koehn

Johns Hopkins University, Baltimore / USA

Multilingual Language Models Language Model Steering LLM Inference Inference-Time Adaptation of Language Models Cross-Lingual Transfer

Ask PDF

Abstract

Summarization Evaluation

Explore similar work

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference