Visual demonstrations provide a natural interface for specifying image transformations that are difficult to describe exhaustively with text. However, existing visual in-context learning (VICL) methods primarily focus on appearance-level relation transfer and provide limited support for physically grounded transformations, whose outcomes depend on material properties, geometry, object interactions, and environmental conditions. Given a source--target exemplar pair and a query image, physically grounded VICL requires a model to infer the demonstrated transformation, adapt its effects to the query-specific scene context, and preserve rule-irrelevant content. We introduce PhysVICL-74, comprising 74 physically grounded transformation rules and 5,240 source--target image pairs that form nearly 75K training and evaluation contexts. Its benchmark split separately evaluates novel-instance transfer and unseen-rule generalization. We further propose TransPhy, a framework that decomposes physically grounded VICL into physical-rule induction and transition-aligned rendering. TransPhy first predicts the demonstrated rule and an explicit query-specific target-state description, and then synthesizes the target image through token-wise mixture-of-experts adaptation, with expert routing guided by localized transition cues. Experiments show that TransPhy improves physical-rule adherence, query consistency, and unseen-rule generalization over existing visual in-context editing methods.
Figures & tables
Figure 1: Physically grounded VICL. Top: limitations of prior methods (1) and our TransPhy framework (2). Bottom: representative PhysVICL-74 transformations.
Figure 2: Overview of TransPhy on BAGEL. The understanding and generation pathways encode the exemplar–query context; ViT-derived transition evidence aligns token-level routing, while MoE-LoRA experts render the inferred physical rule.
Figure 3: Qualitative comparison on seen and unseen transformations. Left: new instances of seen transformations. Right: unseen transformations.
FLUX.1-Fill-dev
BAGEL-MoE
TransPhy
Metric
Scene-Level
Object-Level
Matter-Level
Scene-Level
Object-Level
Matter-Level
Scene-Level
Object-Level
Matter-Level
TA ↑
3.45
3.12
3.33
3.22
3.03
3.63
3.57
3.39
3.73
CP ↑
3.23
3.18
3.41
3.12
3.57
3.23
3.43
3.78
3.67
RP ↑
3.20
3.67
3.09
3.33
3.21
3.28
3.47
3.36
3.25
CLIP-D ↑
0.20
0.25
0.25
0.21
0.27
0.29
0.23
0.26
0.27
LPIPS ↓
0.39
0.36
0.26
0.40
0.36
0.26
0.31
0.31
0.24
Table 1: Seen-rule transfer to novel instances across three rule families. Scene-Level, Object-Level, and Matter-Level denote Scene-Condition, Mechanically Induced, and Material-State transformations, respectively. TA, CP, and RP are scored by GPT-5.6 ; CLIP-D and LPIPS are objective metrics. Results are macro-averaged over rules within each family.
Method
TA ↑
CP ↑
RP ↑
CLIP-D ↑
LPIPS ↓
FLUX.1-Fill-dev
2.63/1.60
3.45/3.71
3.22/1.79
0.24/0.26
0.51/0.56
BAGEL-MoE
2.55/2.16
3.60/2.96
3.24/1.90
0.24/0.23
0.54/0.61
TransPhy
2.91 /2.34
3.75 /3.05
3.28/ 2.90
0.31 /0.28
0.48 / 0.50
RelationAdapter
2.60/3.21
3.17/3.88
3.34/2.82
0.20/0.37
0.52/0.52
VisualCloze
2.76/2.10
3.15/3.41
3.04/2.45
0.25/0.22
0.51/0.54
LoRWeB
1.41/2.16
3.71/2.63
3.44/2.43
0.12/0.26
0.58/0.53
Table 2: Generalization to transformations unseen during TransPhy training. Results are reported as PhysVICL-74/Relation252K over 17 held-out PhysVICL-74 rules and 20 sampled Relation252K rules, respectively.
Variant
TA ↑
CP ↑
RP ↑
CLIP-D ↑
LPIPS ↓
Full model ( E=4 )
3.25
3.46
3.36
0.27
0.31
W/o Stage 1
1.78
3.08
1.87
0.21
0.39
W/o STC align.
3.18
3.24
2.95
0.26
0.37
E=8
3.28
3.46
3.66
0.29
0.29
E=16
3.34
3.52
3.74
0.29
0.30
Table 3: Ablation of training design and expert count. The full model uses staged training, STC alignment, and four experts. The first two variants modify the training design; the last two change only the expert count.
Metric
ρ=5%
ρ=15%
ρ=30%
Token precision ↑
84.18
69.19
41.65
Token recall ↑
28.25
65.03
72.57
Token F1 ↑
42.30
67.05
52.92
Table 4: Transition-token selection quality. Each ρ is evaluated on the fixed subset after connected-component merging and feature-similarity refinement.
Figure 4: Transition-token visualization across ρ .
Figure 5: Expert routing visualization. Left: layer-wise expert-routing heat map. Right: results obtained by hard-routing all tokens through experts 0–3.
Instruction-based image editing has achieved remarkable success in semantic alignment, yet state-of-the-art models frequently fail to render physically plausible results when editing involves complex causal dynamics, such as refraction or material deformation. We attribute this limitation to the dominant paradigm that treats editing as a discrete mapping between image pairs, which provides only boundary conditions and leaves transition dynamics underspecified. To address this, we reformulate physics-aware editing as predictive physical state transitions and introduce PhysicTran38K, a large-scale video-based dataset comprising 38K transition trajectories across five physical domains, constructed via a two-stage filtering and constraint-aware annotation pipeline. Building on this supervision, we propose PhysicEdit, an end-to-end framework equipped with a textual-visual dual-thinking mechanism. It combines a frozen Qwen2.5-VL for physically grounded reasoning with learnable transition queries that provide timestep-adaptive visual guidance to a diffusion backbone. Experiments show that PhysicEdit improves over Qwen-Image-Edit by 5.9% in physical realism and 10.1% in knowledge-grounded editing, setting a new state-of-the-art for open-source methods, while remaining competitive with leading proprietary models.
Language models switch tasks from demonstrations, but vision still often relies on separate systems for restoration, editing, and stylization. A before--after example offers a shared instruction only if the transferable change can be separated from the scene that demonstrates it. We introduce CrossPair, which adapts a pretrained multimodal image editor to transfer diverse transformations from one exemplar pair. Exemplar-before, exemplar-after, and query images take fixed roles in its native VLM and VAE pathways: the VLM extracts operation semantics, while the VAE retains fine appearance and structure. A training-only, change-aware DINOv3 objective aligns feature-change directions across content-distinct scenes, guiding the output to follow the demonstrated relation without an inference-time encoder. We construct Transform-360K by grouping filtered pairs from conditional representations, restoration, editing, virtual try-on, and stylization under shared transformations. On Relation252K-derived seen and held-out transformations, \method improves target agreement and edit-direction similarity over released systems. Ablations show that the semantic and visual exemplar routes are complementary and that DINO relation alignment further strengthens transfer; Transform-360K training also improves aggregate held-out-operation scores.
Yibo Zhao, Lifan Jiang, Senlin cheng +7
State Key Lab of CAD&CG, Zhejiang University · Ant Group · School of Software Technology, Zhejiang University
While instruction-based image editing, enabled by multi-modal generative models, has advanced significantly, existing benchmarks lack a comprehensive evaluation of physics-based reasoning, a critical capability for handling real-world scenarios. To address this, we introduce PhyEditBench, a benchmark designed to assess the physical understanding of editing models. Guided by a hierarchical taxonomy, we establish 4 primary classes and 12 subclasses. It comprises 238 high-quality, high-resolution, real-world instances meticulously extracted from videos to capture authentic physical dynamics, alongside 35 synthetic Anti-Physics instances. Our empirical analysis of current SOTA editing methods exposes substantial limitations in their physics-based reasoning. We further propose a training-free baseline named PhyWorld that uses test-time scaling and a latent reduction strategy. PhyWorld outperforms comparable models and suggests that the video generation process can effectively serve as a reasoning mechanism for image editing. The project page is available at https://github.com/Previsior/PhyEditBench.