From Tokens to Trajectories: Parameter-Efficient Continuous Adaptation of Unified Multimodal Models

Gayatri Deshmukh; Onkar Susladkar; Adheesh Juvekar; Halil Kilicoglu; Ismini Lourentzou

cs.CVNov 3, 2026

From Tokens to Trajectories: Parameter-Efficient Continuous Adaptation of Unified Multimodal Models

Authors: Gayatri Deshmukh, Onkar Susladkar, Adheesh Juvekar, Halil Kilicoglu, Ismini Lourentzou

Organizations: University of Illinois Urbana-Champaign

Abstract

While discrete tokenization in Unified Multimodal Models (UMMs) establishes strong semantic reasoning, it introduces a strict quantization bottleneck that degrades fine-grained visual details, often resulting in structurally accurate but artifact-heavy images. We introduce FlowShift, a parameter-efficient post-training framework that bypasses this bottleneck by converting pre-trained discrete UMMs into continuous rectified-flow generators. Instead of discarding the backbone's semantic priors, FlowShift replaces discrete visual interfaces with continuous latent projections and applies LoRA to the attention layers. To efficiently boost visual quality without semantic drift, we propose Dual-Objective Velocity Vector Distillation (DOVD), which aligns the model with a high-fidelity continuous teacher and ground-truth continuous trajectories. This approach drastically improves perceptual quality while preserving prompt adherence and reasoning. When applied to leading discrete UMMs, FlowShift significantly improves COCO-30K FID and aesthetic scores, while retaining over 99.5% of multimodal understanding capabilities. Ablations confirm DOVD drives generation quality. Ultimately, FlowShift achieves high-fidelity continuous generation without sacrificing pre-trained semantic grounding.

Figures & tables

Figure 2: Discrete vs. continuous generative transport: discrete models jump between categorical states, while Flow Matching follows smooth ODE trajectories from prior to target.

Figure 3: FlowShift : a frozen discrete UMM is adapted to continuous latent flow using lightweight trainable modules and dual-objective velocity distillation from a frozen continuous teacher.

Baselines	Params (B)	Inf. Time (s)	GenEval [ 7 ]	COCO-30K [ 18 ]		Aesthetic-Eval [ 58 ]
Baselines	Params (B)	Inf. Time (s)	Accuracy $\uparrow$	FID $\downarrow$	DINOv2-FID $\downarrow$	FID $\downarrow$	CLIPScore $\uparrow$	Aesthetic Score $\uparrow$
Continuous Flow Matching (CFM) based Native Image Generation Models
Flux1 [ 13 ]	14	16	0.66	9.15	10.43	39.41	33.42	5.51
FLUX.2 [klein] 9B [ 14 ]	9	12	0.75	8.92	9.17	38.47	35.41	6.28
Qwen-Image RL [ 44 ]	22	21	0.88	8.55	9.32	39.09	33.98	6.43
Flux-2 32B [ 14 ]	32	33	0.89	7.12	7.77	33.14	39.21	7.56

Table 1: Image generation performance, model size, and inference time. Subscript values denote the score change of each FlowShift model relative to its corresponding DFM source model.

Continuous Flow Matching (CFM) based Unified Models
Baselines	MMBench [ 21 ]	MMMU [ 57 ]	MM-VET [ 56 ]	MathVista [ 23 ]	MMVP [ 39 ]
BAGEL [ 5 ]	85.00	55.30	67.20	73.10	69.30
OmniGen2 [ 46 ]	79.10	53.10	61.80	69.42	68.26
MammothModa-2 [ 28 ]	86.60	71.23	79.40	81.80	77.50
Discrete Flow Matching (DFM) based Unified Models
MUDDIT [ 31 ]	55.48	49.87	40.14	70.15	66.66

Table 2: Multimodal understanding performance. Subscript values beside FlowShift models denote the average percentage of source-model performance retained after conversion.

Figure 4: Qualitative comparison of discrete and continuous generation with UniDFlow-4B.

Figure 5: Qualitative comparison of discrete and continuous generation of FUDOKI.

Variants	Image Generation				Semantic Understanding
	GenEval	COCO-30K	Aesthetic-Eval		MMBench	MMMU
	Accuracy $\uparrow$	FID $\downarrow$	CLIP Score $\uparrow$	Aesthetic Score $\uparrow$	Accuracy $\uparrow$	Accuracy $\uparrow$
Main Baseline (Qwen as teacher, LoRA adapters, and UniDFlow as base unified model)
Main Baseline	0.92	8.92	38.23	6.21	90.92	74.26
Impact of Distillation
No Teacher Model	0.89 -0.03	10.69 +1.77	36.31 -1.92	5.58 -0.63	89.14 -1.78	73.28 -0.98

Table 3: Ablation study on the impact of distillation, teacher models, and parameter-efficient adaptation. Subscript values denote changes relative to the Main Baseline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Image Generation
Benchmark	# Examples	Format	Task	Evaluation Metric
GenEval [ 7 ]	553	Text Prompt	Compositional T2I	GenEval Acc. $\uparrow$
COCO-30K [ 18 ]	30,000	Image Caption	Text-to-Image	FID $\downarrow$ , DINOv2-FID $\downarrow$
Aesthetic-Eval [ 58 ]	2,781	Image–Text Pair	High-Resolution T2I	FID $\downarrow$ , CLIPScore $\uparrow$ , Aesthetic Score $\uparrow$
Multimodal Understanding
MMBench v1.1 [ 21 ]	3,217	Multiple Choice	General Multimodal QA	CircularEval Acc. $\uparrow$

Appendix

Table 4: Summary of evaluation benchmarks. We report the number of evaluation examples, input–output format, task category, and evaluation metric. For image-generation benchmarks, each example denotes a text prompt, while for multimodal-understanding benchmarks, each example corresponds to a visual question-answering instance. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values are preferred.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 303 papers

Latent Reasoning in Continuous Space for Unified Multimodal Models
Oct 30, 2026Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee +2Continuous Latent Reasoning Unified Multimodal Models
KAIST · POSTECH · RLWRLD
Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal Models Multimodal Pretraining
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics
UniRect-CoT: Enhancing Generation in Unified Multimodal Models via Reflective Rectification with Inherent Understanding
Apr 15, 2026Yibo Jiang, Tao Wu, Rui Jiang +4Diffusion Model Guidance Unified Multimodal Models
School of Software Technology, Zhejiang University · College of Computer Science and Technology, Zhejiang University

Oct 30, 2026J/K move · Enter open · S save

Latent Reasoning in Continuous Space for Unified Multimodal Models

Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho+1

KAIST · POSTECH · RLWRLD

Continuous Latent Reasoning Unified Multimodal Models Compositional T2I Generation RL for Image Generation

Open paper AskPDF arXiv

Unified Multimodal Models (UMMs) achieve remarkable success on diverse multimodal understanding and generation tasks, yet they still struggle to effectively leverage language-based chain-of-thought reasoning for image generation. As a result, existing reasoning-augmented UMMs often fail to inherit the test-time scaling and fine-grained controllability observed in large language models. We attribute this failure to a fundamental modality mismatch: reasoning is performed in a discrete token space, whereas image generation operates in a continuous space. To bridge this gap, we introduce a Latent Reasoning framework in Continuous space (LARC), which alleviates the token-level bottleneck by reasoning over continuous hidden states. Specifically, the training framework consists of two stages: (i) curriculum supervised fine-tuning (SFT), and (ii) information-gain reinforcement learning (RL). First, the curriculum SFT stage gradually converts token-level reasoning into latent reasoning. Subsequently, the information-gain RL stage self-evolves latent reasoning without ground truth supervision. Experiments across diverse text-to-image generation benchmarks show that LARC consistently improves generation quality, prompt following, and compositional fidelity. Notably, LARC achieves state-of-the-art performance on GenEval with a 7%p gain over BAGEL and demonstrates clear test-time scaling behavior, outperforming token-based generation under both parallel and sequential test-time scaling strategies.

Abstract

Unified Multimodal Models

Explore similar work

Latent Reasoning in Continuous Space for Unified Multimodal Models

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

UniRect-CoT: Enhancing Generation in Unified Multimodal Models via Reflective Rectification with Inherent Understanding