PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation

Xiaowei Su; Xusen Cheng

cs.CLOct 12, 2026

PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation

Authors: Xiaowei Su, Xusen Cheng

Organizations: Renmin University of China, Beijing, China · Beijing Bayuegua Technology Co., Ltd., Beijing, China

Abstract

We present PatTransBench, a 12,000-item Chinese-English/Japanese/Korean patent machine translation (PatMT) benchmark with a four-dimensional evaluation protocol. The benchmark comprises 6,000 short-text (sentence/claim) and 6,000 long-text (paragraph) pairs, evenly split across three language pairs and both translation directions, built via IPC-section-stratified sampling and dualthreshold quality gating from upstream patent-family parallel corpora. The evaluation protocol spans literal n-gram overlap (BLEU, ROUGE-L, chrF), cross-lingual semantic textual similarity (CLSTS), 38-probe-language consistency testing, and serving performance under concurrency. A case study on PatMT2-7B and its fp8/AWQ4/AutoRound-4bit variants reveals direction-dependent quantization effects masked by aggregate scores and non-monotonic language-failure patterns across quantization schemes that are invisible to single-metric evaluation. We plan to release the benchmark, protocol, and case-study results upon publication to support reproducible PatMT evaluation.

Figures & tables

Figure 1: Per-IPC-section item counts. Short-text (left): exact 250-item uniform coverage across all eight sections. Long-text (right): skewed to A–C (zh-en/zh-ko) and A–F (zh-ja) after dual-threshold quality gating.

Model	Dir.	BLEU	ROUGE-L	chrF
base	zh $\to$ en	40.31	72.70	68.04
	en $\to$ zh	42.96	54.31	42.21
	zh $\to$ ja	48.65	13.02	53.34
	ja $\to$ zh	48.38	39.63	49.03
	zh $\to$ ko	36.44	42.48	44.56
	ko $\to$ zh	43.59	40.54	41.49

Table 1: BLEU / ROUGE-L / chrF by variant and direction (4,000 items/direction).

Lang.	Chars	Time(s)	Failure
Tibetan	18,140	177.4	Phrase repeated 464 $\times$
Mongolian	9,798	177.4	Token loop
Tamil	14,020	137.6	“Moreover” repeated 1,949 $\times$
Uyghur	45	0.3	Instruction echo

Table 2: Language-consistency failures (BF16/fp8, representative). Failure sets vary across quantization variants; see text for details.

Figure 2: QPS and TPS vs. concurrency, zh

\to

en short-text, fp8 vs. awq4. Both scale near-linearly; fp8 leads at every concurrency level.

Var.	Type	Conc.	QPS	TPS	TTFT avg
fp8	short	10	55.47	4995	0.018s
	short	20	102.79	9207	0.019s
	short	50	208.79	18760	0.023s
	long	10	1.97	3966	0.073s
	long	20	3.02	6115	0.079s
	long	50	4.51	9077	0.101s

Table 3: Serving performance, zh

\to

en, fp8 vs. awq4 (2,000 items/cell).

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

PATMT2: Two-stage domain adaptation for multilingual patent translation
Oct 12, 2026Xiaowei Su, Xusen ChengMachine Translation
Renmin University of China, Beijing, China · Beijing Bayuegua Technology Co., Ltd., Beijing, China
PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks
May 16, 2026Máté Metzger, Nadnapang PhophichitLLM Evaluation Machine Translation
Independent Researcher · International Buddhist Studies College · Mahachulalongkornrajavidyalaya University
Last Translation Benchmark
Sep 3, 2026Vilém Zouhar, Niyati Bafna, Mukund Choudhary +257Multimodal MT MT Evaluation

Oct 12, 2026J/K move · Enter open · S save

PATMT2: Two-stage domain adaptation for multilingual patent translation

Xiaowei Su, Xusen Cheng

Renmin University of China, Beijing, China · Beijing Bayuegua Technology Co., Ltd., Beijing, China

Machine Translation

Open paper AskPDF arXiv

We present PatMT2 (Patent Machine Translation, v2), a two-stage LoRA pipeline that adapts the Hy-MT2-7B dedicated translation model to four-language (Chinese/English/Japanese/Korean) patent translation. On six translation directions, the Stage-2 checkpoint reaches a macro-average BLEU of 43.39 (BF16) to 43.57 (AWQ INT4), improving over the Hy-MT2-7B base by up to +8.8 BLEU and also exceeding Qwen3-32B and several external systems on the same average. We then measure the cost of adaptation: probing 38 languages absent from the fine-tuning data, the model still produces correct, fluent output on 34, while four low-resource languages collapse into distinct, reproducible failure modes. We treat these as diagnostic cases rather than retention rates, and report their mechanisms without attributing them to a single cause. Paired bootstrap tests confirm the translation gains over all baselines (p < 0.001).

Abstract

MT Evaluation

Explore similar work

PATMT2: Two-stage domain adaptation for multilingual patent translation

PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks

Last Translation Benchmark