Visual demonstrations provide a natural interface for specifying image transformations that are difficult to describe exhaustively with text. However, existing visual in-context learning (VICL) methods primarily focus on appearance-level relation transfer and provide limited support for physically grounded transformations, whose outcomes depend on material properties, geometry, object interactions, and environmental conditions. Given a source--target exemplar pair and a query image, physically grounded VICL requires a model to infer the demonstrated transformation, adapt its effects to the query-specific scene context, and preserve rule-irrelevant content. We introduce PhysVICL-74, comprising 74 physically grounded transformation rules and 5,240 source--target image pairs that form nearly 75K training and evaluation contexts. Its benchmark split separately evaluates novel-instance transfer and unseen-rule generalization. We further propose TransPhy, a framework that decomposes physically grounded VICL into physical-rule induction and transition-aligned rendering. TransPhy first predicts the demonstrated rule and an explicit query-specific target-state description, and then synthesizes the target image through token-wise mixture-of-experts adaptation, with expert routing guided by localized transition cues. Experiments show that TransPhy improves physical-rule adherence, query consistency, and unseen-rule generalization over existing visual in-context editing methods.
Figures & tables
Figure 1: Physically grounded VICL. Top: limitations of prior methods (1) and our TransPhy framework (2). Bottom: representative PhysVICL-74 transformations.
Figure 2: Overview of TransPhy on BAGEL. The understanding and generation pathways encode the exemplar–query context; ViT-derived transition evidence aligns token-level routing, while MoE-LoRA experts render the inferred physical rule.
Figure 3: Qualitative comparison on seen and unseen transformations. Left: new instances of seen transformations. Right: unseen transformations.
FLUX.1-Fill-dev
BAGEL-MoE
TransPhy
Metric
Scene-Level
Object-Level
Matter-Level
Scene-Level
Object-Level
Matter-Level
Scene-Level
Object-Level
Matter-Level
TA ↑
3.45
3.12
3.33
3.22
3.03
3.63
3.57
3.39
3.73
CP ↑
3.23
3.18
3.41
3.12
3.57
3.23
3.43
3.78
3.67
RP ↑
3.20
3.67
3.09
3.33
3.21
3.28
3.47
3.36
3.25
CLIP-D ↑
0.20
0.25
0.25
0.21
0.27
0.29
0.23
0.26
0.27
LPIPS ↓
0.39
0.36
0.26
0.40
0.36
0.26
0.31
0.31
0.24
Table 1: Seen-rule transfer to novel instances across three rule families. Scene-Level, Object-Level, and Matter-Level denote Scene-Condition, Mechanically Induced, and Material-State transformations, respectively. TA, CP, and RP are scored by GPT-5.6 ; CLIP-D and LPIPS are objective metrics. Results are macro-averaged over rules within each family.
Method
TA ↑
CP ↑
RP ↑
CLIP-D ↑
LPIPS ↓
FLUX.1-Fill-dev
2.63/1.60
3.45/3.71
3.22/1.79
0.24/0.26
0.51/0.56
BAGEL-MoE
2.55/2.16
3.60/2.96
3.24/1.90
0.24/0.23
0.54/0.61
TransPhy
2.91 /2.34
3.75 /3.05
3.28/ 2.90
0.31 /0.28
0.48 / 0.50
RelationAdapter
2.60/3.21
3.17/3.88
3.34/2.82
0.20/0.37
0.52/0.52
VisualCloze
2.76/2.10
3.15/3.41
3.04/2.45
0.25/0.22
0.51/0.54
LoRWeB
1.41/2.16
3.71/2.63
3.44/2.43
0.12/0.26
0.58/0.53
Table 2: Generalization to transformations unseen during TransPhy training. Results are reported as PhysVICL-74/Relation252K over 17 held-out PhysVICL-74 rules and 20 sampled Relation252K rules, respectively.
Variant
TA ↑
CP ↑
RP ↑
CLIP-D ↑
LPIPS ↓
Full model ( E=4 )
3.25
3.46
3.36
0.27
0.31
W/o Stage 1
1.78
3.08
1.87
0.21
0.39
W/o STC align.
3.18
3.24
2.95
0.26
0.37
E=8
3.28
3.46
3.66
0.29
0.29
E=16
3.34
3.52
3.74
0.29
0.30
Table 3: Ablation of training design and expert count. The full model uses staged training, STC alignment, and four experts. The first two variants modify the training design; the last two change only the expert count.
Metric
ρ=5%
ρ=15%
ρ=30%
Token precision ↑
84.18
69.19
41.65
Token recall ↑
28.25
65.03
72.57
Token F1 ↑
42.30
67.05
52.92
Table 4: Transition-token selection quality. Each ρ is evaluated on the fixed subset after connected-component merging and feature-similarity refinement.
Figure 4: Transition-token visualization across ρ .
Figure 5: Expert routing visualization. Left: layer-wise expert-routing heat map. Right: results obtained by hard-routing all tokens through experts 0–3.