FlowTranslator: Training-Free In-Image Machine Translation with Flow-Based Editing

Zhixian Li; Xingsong Ye; Yongkun Du; Yunyan Xiao; Yanglin Huang; Pinxuan Chen; Jiyu Zhuang; Cong Xu

cs.CVNov 3, 2026

FlowTranslator: Training-Free In-Image Machine Translation with Flow-Based Editing

Authors: Zhixian Li, Xingsong Ye, Yongkun Du, Yunyan Xiao, Yanglin Huang, Pinxuan Chen, Jiyu Zhuang, Cong Xu,

Organizations: Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI

Abstract

In-image machine translation (IIMT) replaces text in an image with its translation while preserving the surrounding visual context. Existing task-specific models achieve convincing performance but require massive paired training images before and after translation. As advanced text-to-image (T2I) models have reached a practically usable level of multilingual text generation, we argue that this capability can be transferred to IIMT and present FlowTranslator. It is a training-free IIMT framework that first employs an OCR model and an LLM to generate the translation prompt, which is injected into the T2I-based diffusion model. To optimize the accurate and natural generation of translated text, we introduce two trajectory controls: Region Coupling Release (RCR) partially decorrelates target-branch noise inside the text region early in the trajectory, while Coupling Expansion (CE) amplifies the emerging edit displacement mid-trajectory. Under the EasyOCR read-back protocol, FlowTranslator achieves average PRIM BLEU and COMET scores of 20.6 and 57.2, respectively, substantially outperforming the strong training-based baseline UniTranslator. On the more challenging and realistic whole-image ScenePair photographs translated into Chinese, it reaches 40.1% exact-match accuracy and surpasses existing editing systems, some of which only address the text-rendering stage rather than the complete IIMT task. These results establish flow-based editing as a strong, fully training-free baseline for IIMT.

Figures & tables

Figure 1: Overview of the IIMT framework and editor paradigms. (a) The pipeline: OCR extracts text → a translator produces the translation → a scene text editor generates the output image. (b) Rule-based: remove, render, and paste separately. (c) Training-based: a learned GAN or diffusion editor (requires extra training). (d) FlowTranslator (ours): training-free flow editing with trajectory controls (CE and RCR) for text translation with background preservation.

Figure 2: Overview of FlowTranslator. OCR extracts the source text and forms a source caption, while a frozen LLM translates it to obtain the target caption. A frozen VAE encodes the source image into $Z^{\mathrm{src}}_{0}$ , from which noise initialization produces $Z^{\mathrm{FE}}_{T}$ . At each editing step, the same frozen DiT predicts source- and target-conditioned velocities. Their difference updates the flow-editing latent. RCR perturbs the target-branch noise within the detected text region, and CE amplifies the accumulated displacement before it is fed back to the target branch. The final latent $Z^{\mathrm{FE}}_{0}$ is decoded to produce the translated image.

		BLEU $\uparrow$				Avg.
Model	IIMT-tuned	De $\rightarrow$ En	En $\rightarrow$ De	Fr $\rightarrow$ En	Ro $\rightarrow$ En	BLEU $\uparrow$	SSIM $\uparrow$
GPT-Image-1 [ 20 ]		20.75	19.61	27.38	24.50	23.06	0.734
Qwen-Image [ 31 ]		0.23	0.21	0.20	0.22	0.22	0.785
OCR+MT+T2I [ 14 ]	✓	15.37	13.22	21.60	18.34	17.13	0.771
PEIT+T2I [ 47 ]	✓	10.91	8.77	18.51	14.54	13.18	0.769
Translatotron-V [ 14 ]	✓	15.39	13.23	22.20	18.44	17.32	0.776

Table 1: TTV benchmark with EasyOCR read-back and lowercased sacreBLEU. “IIMT-tuned” marks systems trained or fine-tuned on IIMT data. GPT-Image-1 and Qwen-Image are zero-shot prompted unified models. “Masked FlowEdit” is our baseline with identical hyperparameters and the trajectory controls disabled. Average columns are unweighted means over the four directions. For each metric, best in bold , second underlined .

Figure 3: Qualitative results on TTV. Each row shows the source image, the benchmark target image, and the FlowTranslator output for one translation direction. The examples cover EN $\rightarrow$ DE, FR $\rightarrow$ EN, DE $\rightarrow$ EN, and RO $\rightarrow$ EN.

		BLEU $\uparrow$ / COMET $\uparrow$					Avg.
Model	IIMT-tuned	En-De	En-Fr	En-Cs	En-Ru	En-Ro	BLEU	COMET
PARSeq-mT-Render [ 26 ]	✓	9.5 / 41.7	13.8 / 46.9	7.7 / 43.9	5.5 / 48.1	12.8 / 53.5	9.9	46.8
PEIT-Render [ 47 ]	✓	10.4 / 45.1	14.0 / 48.1	7.9 / 46.2	5.3 / 47.7	14.2 / 52.9	10.4	48.0
Translatotron-V [ 14 ]	✓	1.7 / 34.3	1.9 / 30.2	1.1 / 30.5	0.9 / 32.0	1.3 / 33.9	1.4	32.2
VisTrans [ 26 ]	✓	12.6 / 44.4	17.0 / 49.4	5.9 / 41.8	7.2 / 49.4	13.9 / 50.2	11.3	47.0
UniTranslator [ 18 ]	✓	13.1 / 48.7	21.6 / 58.5	7.4 / 43.3	8.4 / 50.2	13.5 / 52.8	12.8	50.7

Table 2: PRIM benchmark with EasyOCR read-back. BLEU uses both official reference sets, while COMET uses the Google references. Averages are computed from unrounded per-direction scores.

Model	STE-tr.	Zh	ACC $\uparrow$	SSIM $\uparrow$
TextDiffuser [ 3 ]	✓		0.00	0.9434
DiffSTE [ 10 ]	✓		0.23	0.9796
AnyText [ 28 ]	✓	✓	36.80	0.8157
TextFlux [ 34 ]	✓	✓	14.30	0.9506
FLUX.1-Fill [ 2 ]			0.39	0.9231
FLUX.1-Kontext [ 1 ]			0.00	0.4273

Table 3: ScenePair En

\rightarrow

Zh whole-image translation on 1,280 real scenes. All methods share the same text boxes and target translations. “STE-tr.” marks scene-text-editing training and “Zh” native Chinese glyph support. Rows without “Zh” are capability diagnostics rather than head-to-head competitors.

Figure 4: Qualitative comparison on whole-image ScenePair En→Zh translation. For each example, we show the source image (with red boxes marking text to be translated), the requested target text, and outputs of existing open-source methods, followed by FlowTranslator. FlowTranslator renders the requested Chinese text within the original region. It preserves both the surrounding scene and the local text appearance.

Component	Setting	BLEU $\uparrow$
Controls (PRIM avg.)	Masked FlowEdit (Baseline)	16.88
	+RCR only	17.40
	+CE only	20.21
	+RCR+CE (Ours)	20.57
Edit window $n_{\max}$	43	18.95
	45 (Default)	20.57

Table 4: Component ablations. Control rows are full PRIM runs (340 images per direction, averaged over the five directions) sharing the same hyperparameters (

\gamma{=}1.3

, 15% dilation, RCR

\kappa{=}0.5

, CE

c{=}0.4

) and differing only in the mechanism switches.

Figure S1 : RCR diagnostics: dashed/solid pairs nearly overlap at plotting resolution (Masked FlowEdit with

+

CE, and

+

RCR with

+

RCR

+

CE). (a) In-mask velocity alignment. Bands show the min–max range over 12 images for Masked FlowEdit and

+

RCR. Dotted line:

\tau_{\mathrm{lo}}=0.3

. (b) In-mask edit-signal concentration on a log scale. The first third of the edit window is shaded (

+36\%

). (c) Prescribed versus realized noise correlation (mean absolute error

0.0032

). (d) First-step RCR and CE perturbations by arm. Zeros are explicitly labeled.

Diagnostic	Masked FlowEdit	+RCR	+CE	+RCR+CE
early $\cos_{\rm in}$	0.740	0.481	0.738	0.480
early signal ratio	1.94	2.63	1.93	2.63
ignition step $\downarrow$	9.50	7.75	9.42	7.67
final $d_{\rm in}$	0.440	0.443	0.533	0.513
max rel. CE pert.	0.000	0.000	0.267	0.246
max $\|Z_{t}^{\rm FE}\|$	3.991	4.052	3.991	4.053

Table S1 : Trajectory diagnostics over 12 images (45 edit steps each). Rows are means across images except the two “max” rows, which are maxima over images and steps. “Early” denotes the first third of the edit window. The ignition step uses a single absolute threshold shared by all arms. Dashes mark probes that are inactive in an arm by construction.

Figure S2 : CE effectiveness and stability: pairing as in Fig. S1 . (a) In-mask displacement normalized by the Masked FlowEdit final value. Dashed line: shared 10% ignition threshold. (b) Ignition step with per-image values. (c) Relative CE perturbation. CE-off arms lie at zero. (d) Paired per-step difference in the latent maximum relative to Masked FlowEdit. Global maxima for Masked FlowEdit and

+

RCR

+

CE differ by

1.6\%

Figure S3 : Additional qualitative examples of full-image scene text translation into Chinese. Each pair shows the source image on the left and the corresponding FlowTranslator output on the right.

Image Text Translation

See all 10 papers

UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation
Jun 23, 2026Jiahao Lyu, Pei Fu, Zhenhang Li +6Image Text Translation Unified Multimodal Models
Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China +2
IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation
Mar 11, 2026Jiahao Lyu, Pei Fu, Zhenhang Li +7Image Text Translation Multimodal MT
Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China +3
ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation
Oct 26, 2026Šárka Uramová, Petar Kirilov Petrov, Josef Jon +2Image Text Translation Multimodal Dataset
Charles University, Faculty of Mathematics and Physics, Prague, Czech Republic

Jun 23, 2026J/K move · Enter open · S save

UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation

Jiahao Lyu, Pei Fu, Zhenhang Li, Shaojie Zhang+5

Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China+2

Image Text Translation Unified Multimodal Models Machine Translation Multimodal MT

Open paper AskPDF arXiv

In-Image Machine Translation (IIMT) aims to translate scene text in an image and render the translated text back into the original regions while preserving the overall visual appearance. Recent unified multimodal models provide a promising solution by combining visual-text understanding and image generation within a single framework. However, directly adapting such models to IIMT remains challenging. In particular, they often suffer from understanding-generation conflicts, where the translation inferred during understanding is inconsistent with the text supervision used in generation, and spatial position misalignment, where the rendered text does not accurately match the target text regions. To address these issues, we present UniTranslator, a unified multimodal framework for IIMT that tightly couples translation understanding and text editing. Specifically, we introduce an Understand-Generation Alignment Module (UGAM) to bridge the representation gap between understanding and generation, encouraging semantic consistency between translated content prediction and text rendering. We further propose a Spatial Mask Decoder (SMD) with pixel-level supervision over text regions to improve spatial grounding, geometric alignment, and layout controllability during generation. Extensive experiments on multiple benchmarks demonstrate that UniTranslator achieves state-of-the-art performance across diverse language directions and complex real-world layouts. Moreover, our results reveal a strong mutual reinforcement effect between translation understanding and image generation, highlighting the advantage of unified translation multimodal learning. Code is available at https://github.com/SeerRay-Lab/Unitranslator.

Abstract

Image Text Translation

Explore similar work

UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation

IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation