When Opposite Premises Translate Identically: A Source-Conditioned Diagnostic for Multilingual Commonsense Items

Shuai Wang; Rui Hu

cs.CLOct 17, 2026

When Opposite Premises Translate Identically: A Source-Conditioned Diagnostic for Multilingual Commonsense Items

Authors: Shuai Wang, Rui Hu

Organizations: Zhejiang University

Abstract

Multilingual reasoning comparisons depend on translated premises preserving the relations in their English sources. Automatic translation scores can rank source-target pairs, but an acceptance decision alone cannot certify a counterfactual benchmark field. We test a source-contrast diagnostic for ACCORD premises: reverse the English polarity, translate both arms independently, count identical target outputs, and score each arm against its own source. Across 71 matched question identities at two containing-item depths, five target languages, and three translation checkpoints, Swahili has 63, 113, and 67 exact collisions among 710 pairs per checkpoint. At a human-reference-calibrated threshold, round-trip NLI passes both arms of 3 of the 113 NLLB-3.3B collisions, while LaBSE passes 78. Each collision proves at least one unfaithful translation, but does not identify its arm. These controlled measurements do not establish screening accuracy on spontaneous translation errors or a language-by-hop reasoning effect; the study has no native equivalence labels or answer-model evaluation.

Figures & tables

Figure 1: Opposed ACCORD premises can collapse to the same target text. The first panel locates a premise in a source item. The second shows field translation across the target languages. The third tests the polarity reversal and counts identical Swahili outputs. The fourth gives round-trip NLI and LaBSE decisions against each arm’s generating source at their calibrated thresholds. A dual acceptance certifies at least one accepted unfaithful field without identifying its arm.

Translator	Language	1-hop	2-hop	RT	Direct	LaBSE	QE
NLLB-1.3B	Spanish	1	0	0	0	0	0
NLLB-1.3B	Hindi	1	0	0	0	0	0
NLLB-1.3B	Swahili	52	11	4	0	43	11
NLLB-1.3B	Hausa	7	0	3	7	3	5
NLLB-1.3B	Yoruba	2	0	2	2	1	2
NLLB-3.3B	Spanish	2	3	1	0	0	1

Table 1: Exact target collisions and dual source-conditioned screen acceptance. Each depth and language has 355 opposed premise pairs from 71 questions. Collision counts are target-string identities. The four right columns count pairs for which both contradictory sources and their identical target pass the screen. RT thresholds are recalibrated for each reverse translator; other screens use the same FLORES human-reference threshold. All columns report counts, not translation-error rates.

Figure 2: Exact Swahili target collisions. Each point is the number of identical translations of opposed premises among 355 pairs at the indicated containing-item depth. Whiskers are percentile 95% intervals from 2,000 source-question bootstrap draws, retaining all five premises. The two NLLB checkpoints share a model family; MADLAD-400 is a separate translator family.

Figure 3: Swahili dual acceptance among exact collisions. Both translated arms must pass against their own English sources. Points are observed fractions; whiskers are percentile 95% intervals from 2,000 source-question bootstrap draws at frozen thresholds. Calibration uncertainty is analyzed separately. A dual-accepted collision proves that at least one accepted pair is unfaithful, without identifying which arm. The chart does not estimate screen precision or recall on naturally occurring errors.

Figure 4: Swahili source-format comparison with unchanged-input controls. Points give exact collision rates in two runs of the same NLLB-3.3B checkpoint. Only the first group has modified English sources; batch size also changed between runs. The unchanged two-hop group shows output variability under that execution change.

Screen	ES	HI	SW	HA	YO
Round trip	1.00	0.98	0.98	0.97	0.93
Direct NLI	1.00	0.99	0.99	0.93	0.60
LaBSE	0.99	0.94	0.98	0.94	0.88
COMET-QE	0.83	0.75	0.67	0.52	0.53

Table 2: AUC for an English entailment hypothesis paired with its human translation versus a human translation of a contradiction-labeled hypothesis from the same premise. The 200 source groups are aligned across the five target languages. Higher is better.

Figure 5: Ranking on secondary controls. Left: human-translated entailment versus independently authored contradiction hypotheses. Right: original versus least-similar sibling-target ACCORD choices. Points are AUC; whiskers resample source groups or ACCORD items. The grey line is chance ranking. Neither control labels natural translation errors.

		FLORES threshold		In-domain folds
Language	Screen	Positive	Altered	Positive	Altered
Spanish	Round trip	188	1	178	0
Spanish	Direct NLI	190	0	177	0
Spanish	LaBSE	180	2	179	1
Spanish	COMET-QE	176	111	179	115
Hindi	Round trip	181	5	180	3

Table 3: Aligned XNLI/AfriXNLI decisions under the FLORES development threshold and five-fold in-domain calibration. Each positive and altered-target column has 200 human-translated pairs. Each fold’s in-domain threshold uses only the other four folds’ positive targets. Counts are accepted pairs, not ACCORD validity labels.

Reverse translator	Spanish	Hindi	Swahili	Hausa	Yoruba
NLLB-1.3B	0.162803	0.031874	0.025652	0.006631	0.001748
NLLB-3.3B	0.135743	0.023931	0.028563	0.007938	0.001714
MADLAD-3B	0.446000	0.023056	0.007618	0.004268	0.000551

Table 4: Round-trip NLI thresholds fitted separately to each reverse translator. Each threshold is the highest observed score retaining at least 90 of 100 aligned FLORES development human translations. Values are measured point estimates; target-language field retention is not known.

	Round trip		Direct NLI		LaBSE		COMET-QE
Language	M	N	M	N	M	N	M	N
Spanish	0.98	0.93	1.00	0.94	1.00	0.68	0.96	0.65
Hindi	0.99	0.96	1.00	0.89	1.00	0.65	0.94	0.55
Swahili	0.99	0.90	1.00	0.87	1.00	0.65	0.84	0.54
Hausa	0.97	0.89	0.98	0.78	1.00	0.60	0.75	0.54
Yoruba	0.86	0.68	0.74	0.48	1.00	0.53	0.54	0.54

Table 5: Held-out AUC for distinguishing human references from wrong-row targets (M) and source-negated pairs (N). Higher is better; the negation comparison uses the corresponding 39 unaltered references.

Language	Screen	Premise	Question	Choice
Spanish	Round trip	118	23	102
Spanish	Direct NLI	107	22	33
Spanish	LaBSE	64	23	73
Spanish	COMET-QE	56	10	67
Hindi	Round trip	114	23	116
Hindi	Direct NLI	118	23	117

Table 6: Calibrated ACCORD field acceptance by type. Each language–screen pair has 120 premise fields, 24 questions, and 120 choices; counts are screen decisions without target-language equivalence labels.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Language	Screen	$\tau$	Devtest	Short	Mismatch	Negated	ACCORD
Spanish	Round trip	0.1628	91	94	0	12	10
Spanish	Direct NLI	0.9887	89	81	0	5	0
Spanish	LaBSE	0.8412	82	70	0	26	0
Spanish	COMET-QE	-0.0456	93	96	18	38	0
Hindi	Round trip	0.0319	96	87	0	14	16
Hindi	Direct NLI	0.6415	94	90	0	15	19

Appendix

Table 7: Held-out screen decisions after per-language thresholds are fixed to admit at least 90 of 100 FLORES dev human references. Columns count accepted ordinary devtest references, disjoint short devtest references, deliberately mismatched translations, source-negated references, and full ACCORD renderings. Their denominators are 100, 100, 100, 39, and 24 per language and screen. Threshold scales differ by screen; counts are decisions, not ACCORD meaning labels.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 483 papers

Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL $\rightarrow$ FOL
Sep 27, 2026Pu Suo, Ali EmamiLanguage Model Error Detection Mathematical Reasoning
Emory University
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss
Apr 14, 2026Ronald Skorobogat, Ameya Prabhu, Matthias BethgeMultilingual Language Model Evaluation Language Model Generation Evaluation
Tübingen AI Center, University of Tübingen

Sep 27, 2026J/K move · Enter open · S save

Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL $\rightarrow$ FOL

Pu Suo, Ali Emami

Emory University

Language Model Error Detection Mathematical Reasoning Semantic Parsing

Open paper AskPDF arXiv

A standard pipeline for symbolic reasoning over natural-language problems translates them into first-order logic and invokes a theorem prover. The translation step is the bottleneck: swap "every" for "some" and every inference that follows is corrupted. Yet today's metrics often score more broken translations higher than less broken ones, because BLEU, BERTScore, and Smatch++ reward surface overlap that the worst errors happen to preserve. We introduce SIV, which derives two kinds of probes from the target formula and uses a theorem prover to verify the candidate translation against each. Positive probes are statements the candidate must entail, which detect translations that drop content; contrastive probes are statements the candidate must not entail, which detect translations that assert more than the original. On a controlled pool of perturbed FOLIO translations, the severity of the error accounts for 80% of SIV's score variance, compared with at most 17% for any prior metric. Across six error classes on a disjoint pool, SIV scores the reference above the perturbed candidate in over 99% of pairs. Because each probe is labeled with what it tests, the failure pattern also supplies a labeled error trace, recovering the perturbation class at macro-F1 0.638, nearly double the score-only baseline. On 434 expert-audited real LLM translations, SIV attains the top AUC, uniquely detects and grades expert-labeled major errors, and abstains, rather than mis-scoring, on out-of-vocabulary translations.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL→\rightarrow→FOL

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL $\rightarrow$ FOL