cs.AIMar 7, 2026
SaveVision Language Models Cannot Reason About Physical Transformation
Organizations: University of Michigan · University of California San Diego · University of California Berkeley · Johns Hopkins University ·
Abstract
Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate and evaluate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with actual visual content when performance is balanced across conserving and non-conserving scenarios. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.
Figures & tables
Figure 1: Illustrative Tasks and Frame Selection Pipeline in Conservation Bench .
| Component | Count |
|---|---|
| Core Dataset | |
| Conservation Tasks | 192 |
| Non-conserving Control | 192 |
| Total Videos | 384 |
| Multi-frame Conditions (Factorial) | |
| Extraction Method | 3 |
Table 1: Overview of Multi-image Task Conditions and Evaluation Scale
Figure 2: Overall Performance on ConservationBench . A. Accuracy averaged across conservation tasks and non-conserving controls, compared to strict pairwise accuracy (purple bars), which requires a model to answer both a conservation task and its matched non-conserving control correctly to be counted as correct (Top 30 models; full results in Appendix H ). B. Performance on non-conserving controls in relation to conservation task accuracy across all 112 VLMs.
Figure 3: Transition matrices showing how model predictions shift when visual input is removed ( A. Empty Image Control; B. Text-only Control). Each cell shows the proportion of samples transitioning from a prediction under full visual input (row) to a prediction under the control condition (column). The large values in column 1, rows 2–3 reveal a strong textual prior toward quantity invariance: models shift toward “Conserve” 70% of the time when visual content is removed. Paradoxically, models perform worse on actual conservation tasks with real visual content ( 60%) than without it (85.7%), indicating that visual content merely disrupts the invariance bias rather than enabling genuine transformation reasoning, producing the observed inverse performance across conserving and non-conserving conditions.
Figure 4: Model performance showing main effects by (A) prompt type, (B) number of frames, and (C) frame sampling method. Each panel averages across the other two factors from the full factorial design (4 prompts 5 frame counts 3 extraction methods). The dashed white line connects mean accuracies across conditions to facilitate visual comparison of trends.
Figure 5: Conservation reasoning does not emerge with model scale. Model performance on (left) conservation tasks shows no relationship with parameter count ( ), while (right) non-conserving task accuracy exhibits only modest scaling effects ( ), both evaluated at 7-frame condition across 112 VLMs.
Figure 6: Cross-benchmark correlation between ConservationBench and nine physical reasoning and video understanding benchmarks, demonstrating that conservation reasoning captures a general factor of physical understanding.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | Conservation Task | Non-conserving Control |
|---|---|---|
| Number | Two rows of identical coins are presented in an initial configuration. One row is then spread apart without adding or removing any coins. | Two rows of identical coins are presented in an initial configuration. One row is then spread apart, with one coin added to said row. |
| Length | Two straws of identical length are shown in an initial configuration. One of the straws is then repositioned without altering its actual length. | Two straws of identical length are shown in an initial configuration. One of the straws is then repositioned while its actual length is altered (extendable straws are used). |
| Volume | A fixed volume of liquid is poured from one container into another of a different shape. Although the height changes significantly, the volume remains constant throughout the transformation. | A fixed volume of liquid is poured from one container into another of a different shape. A significant portion of water is left in the original container instead of being completely poured in. |
| Size | A lump of playdough is reshaped from one form (e.g., a ball) into another (e.g., a flattened disc). While the shape and surface features change, the total mass remains the same across both states. | A lump of playdough is reshaped from one form (e.g., a ball) into another (e.g., a flattened disc). A significant portion of playdough is left in the experimenter’s hand without being integrated into the new shape. |
Table 2: Task descriptions for conservation and non-conserving control scenarios across four quantitative properties.
| Prompt Type | Prompt Example |
|---|---|
| Direct Question | Is the number of coins in the upper row the same as in the lower row in the final image? |
| ”Sequential” Prompt | Please process the images below sequentially, and then answer: Is the number of coins in the upper row the same as in the lower row in the final image? |
| CoT Prompt | Please process the images below sequentially. First describe what happens across the images, then answer: Is the number of coins in the upper row the same as in the lower row in the final image? |
| ”Continuous” Prompt | The above images represent a continuous process. Please answer: Is the number of coins in the upper row the same as in the lower row in the final image? |
Table 3: Four different prompt formats used in our benchmark.
Figure 7: Example visual input: A sequence of 7 frames from a number conservation task, showing the initial state, transformation process, and final state.
| Domain | Parameter | Variations |
| Number | P1: Object Type | 2 variants (Uniform, Mixed) |
| P2: Mapping Shift | 2 variants (Lower vs. Upper row moved) | |
| P3: Distance Spread | 2 variants (Near, Far) | |
| P4: Number of Objects | 6 variants (3–8 coins) | |
| Total combinations: | ||
| Length | P1: Object Type | 2 variants (Uniform, Mixed) |
Table 4: Counterbalanced variations of task-irrelevant features. Each unique combination of parameter values yields 48 distinct task instances per domain.
| Rank | Model | Conserve (%) | Non-Conserve (%) | Average (%) | Strict (%) |
|---|---|---|---|---|---|
| 0 | Human Baseline | 98.55 | 98.15 | 98.35 | 96.72 |
| 1 | gemini-2.5-pro | 94.33 | 43.88 | 69.11 | 41.20 |
| 2 | doubao-seed-1.6-vision | 82.99 | 46.42 | 64.71 | 38.22 |
| 3 | gpt-5 | 91.48 | 35.82 | 63.65 | 33.14 |
| 4 | claude-sonnet-4-5 | 75.43 | 45.39 | 60.41 | 37.23 |
| 5 | Qwen-3-VL-32B-Instruct | 64.57 | 52.06 | 58.31 | 31.15 |
Table 5: Complete Model Rankings by Average Accuracy (Ranks 1-56)
| Rank | Model | Conserve (%) | Non-Conserve (%) | Average (%) | Strict (%) |
|---|---|---|---|---|---|
| 57 | InternVL-3-8B | 65.55 | 16.74 | 41.14 | 8.12 |
| 58 | InternVL-3.5-4B | 59.04 | 22.43 | 40.73 | 8.73 |
| 59 | Phi-3-vision-128k-instruct | 57.61 | 23.54 | 40.58 | 6.28 |
| 60 | InternVL-2.5-4B | 72.66 | 8.06 | 40.36 | 4.66 |
| 61 | Qwen-2.5-Omni-7B | 66.48 | 14.11 | 40.30 | 7.84 |
| 62 | VLAA-Thinker-Qwen2.5VL-7B | 54.77 | 25.56 | 40.16 | 10.56 |
Table 6: Complete Model Rankings by Average Accuracy (Ranks 57-112, cont.)
Figure 8: Average model performance by quantitative domains. Models consistently perform worse on non-conserving controls compared to conservation tasks.
Figure 9: Model performance under concept explanation prompting across four quantitative properties.
Figure 10: Model performance under caption-only conditions across four quantitative properties.
Figure 11: Model performance under video+caption conditions across four quantitative properties.
Figure 12: Model-level correlations between conservation task performance and control condition biases. (Left) Conservation task accuracy versus Empty Image Control accuracy shows strong positive correlation ( , ), indicating models performing better on conservation tasks exhibit stronger textual priors favoring quantity invariance when visual content is removed. (Right) Conservation task accuracy versus Text Control accuracy shows similar but slightly weaker correlation ( , ), with both patterns demonstrating that success on conservation tasks is driven primarily by textual biases rather than visual transformation reasoning. Evaluated on 62 VLMs supporting both empty image and text-only inputs under 7-frame conditions.
Figure 13: Correlation between conservation task accuracy and control condition accuracy with an added “I don’t know” response option ( models).
Figure 14: Prompt type, frame count, and extraction method effects restricted to the top 15 models by average accuracy.
Figure 15: Performance of Cosmos Reason on ConservationBench across four quantitative properties.
Figure 16: Decision confidence (log-probability margin between top-1 and top-2 answer choices) for four prediction categories on Qwen2.5-VL-7B-Instruct. NC_same_error : incorrect “same” prediction on non-conserving trials; NC_correct : correct prediction on non-conserving trials; C_correct : correct prediction on conserving trials; NC_other_error : other incorrect predictions on non-conserving trials.
Figure 17: Layer-wise frame-attention heatmaps for Qwen2.5-VL-7B-Instruct. Top: Non-conserving (NC) trials showing absolute attention distributions for NC_correct, NC_same_error, and NC_non_same_error (top row), alongside pairwise difference maps (bottom row). Bottom: Conserving (C) trials showing C_correct and C_non_same_error with their difference map. The key finding is visible in NC_same_error NC_correct: NC_same_errors place disproportionately more attention on Frame 1 (initial state) in late layers, a pattern absent in all other difference maps.
Figure 18: Example failure case from claude-sonnet-4-5 on a non-conserving volume task. The model performs frame-by-frame perceptual matching rather than reasoning about the transformation as a continuous physical process.
More in this topic
Physical Reasoning
Physical reasoning studies how AI systems infer object properties, forces, and causal dynamics to predict and act in the physical world.
See all 69 papers