cs.CVApr 3, 2026
SaveToken Warping Helps MLLMs Look from Nearby Viewpoints
Organizations: KAIST
Abstract
Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.
Figures & tables
Figure 2: Image Tokenization in MLLMs (Sec. 3.1 ). MLLMs process images by dividing them into fixed-size patches, embedding each patch, and passing them through a vision encoder ( e.g . , ViT) to obtain image tokens.
Figure 3: Limitations of Pixel-Wise Warping. Pixel-wise warping to a target viewpoint often introduces local distortions and semantic degradation. In both forward (top) and backward (bottom) warping, the book from the source view appears significantly distorted after transformation (in the red box).
Figure 4: Pixel-Wise vs. Token Warping. Comparison of inverse warping strategies (Sec. 3.3 ). (A) Pixel-wise warping retrieves pixels for each target coordinate, but patchifying the warped image introduces local distortions, resulting in degraded MLLM understanding. (B) Token warping directly retrieves intact tokens (or patches) from the source view, preserving semantics and improving viewpoint-aware perception.
Figure 5: Fetching Position Noise Sensitivity (Sec. 3.2 ). Through a toy experiment on CV-Bench-2D [ 93 ] , where we emulate local positional perturbations and degradation introduced by warping, we find that token representations in MLLMs are highly robust to noise in the image positions from which tokens are fetched. This suggests that tokens are well suited for representing viewpoint changes.
Figure 6: ViewBench . Example source-target image pairs with corresponding questions and answers from our ViewBench benchmark. The tasks evaluate MLLM’s ability to infer spatial relationships from nearby viewpoints ( Text , Shape ), while also measuring robustness to view changes by asking to describe object properties visible in the warped target view ( Object ).
Figure 7: Token Fetching Strategies. (A) Nearest fetching selects the closest existing token from the source image grid. (B) Adaptive fetching dynamically crops a patch centered at the mapped coordinate to derive a token precisely centered at the target location.
Figure 8: Warping Visualizations. We compare the warped results of pixel-wise warping, token warping, and the generative NVS output [ 85 ] . The rightmost image shows the ground-truth target viewpoint. For token warping, we visualize the RGB image patches corresponding to each token for illustration only. Above each row, we provide the question from ViewBench , and below each image we show the response from Qwen2.5-VL [ 6 ] when given the corresponding warped result. † The camera motion from the source view to the target view is additionally supplied as part of the prompt.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| GT | Depth | Depth & Pose | Ref. | |||
|---|---|---|---|---|---|---|
| DA-V2 | DP | VGGT | DUSt3R | |||
| Pixel-Wise Warp. | 62.35 | 60.49 | 62.76 | 63.58 | 61.29 | 31.48 |
| Token Warp. | 70.99 | 65.84 | 67.74 | 68.95 | 65.05 | |
Table A2: Robustness to Estimated Geometry. Accuracy (%) on ViewBench-Shape (averaged across all overlap levels). Ref. is a no-warping baseline with base Qwen2.5-VL [ 6 ] .
| Depth | GT | Pred. |
|---|---|---|
| Qwen2.5-VL [ 6 ] | 34.39 | – |
| Pixel-Wise Warp. | 61.90 | 61.38 |
| Token Warp. | 65.08 | 66.14 |
Table A3: Larger Viewpoint Shift (2–5% Overlap). Accuracy (%) on a stress-test split with extremely low view overlap, where the source and target views share only 2–5% of visible scene content.
| Depth | GT |
|---|---|
| Qwen2.5-VL [ 6 ] | 32.00 |
| Pixel-Wise Warp. | 38.00 |
| Token Warp. | 46.00 |
Table A4: Occlusion Evaluation. Accuracy (%) on a ProcTHOR-based [ 22 ] split where the queried object is fully occluded in the target view. Token warping consistently outperforms pixel-wise warping and the base Qwen2.5-VL [ 6 ] .
Figure A1 : Occlusion Evaluation. Example source–target pairs from the ProcTHOR-based [ 22 ] occlusion split, where a visible object in the source view becomes fully occluded in the target view.
| Text (%) | Shape (%) | |||||
|---|---|---|---|---|---|---|
| View Overlap (%) | 5–15 | 15–25 | 25–35 | 5–15 | 15–25 | 25–35 |
| Oracle (Geometry) | 93.78 | 93.81 | 93.64 | 95.26 | 94.54 | 93.75 |
Table A5: Geometry-Based Oracle. Accuracy (%) of a geometry-only baseline that determines left–right ordering by comparing -coordinates of the warped source keypoints.
Figure A2 : Qualitative Sample 1. Given the source image (leftmost), the question asks for the spatial relationship between the photo frame (blue box) and the pillow (red box) as viewed from the target viewpoint (rightmost). To visualize tokens, we color-code each source token by its position in the source image, and preserve this color after warping, so the color of each token in the warped views indicates its source location. With forward token warping , the projected tokens become sparse and irregular, leading the MLLM to answer incorrectly. In contrast, backward token warping with nearest fetching produces a dense, regular target token grid, allowing the model to correctly infer the spatial relationship from the target view. (Source and target images are from ARKitScenes [ 8 ] .)
Figure A3 : Qualitative Sample 2. Given the source image (leftmost), the question asks for the order of the toys from left to right as seen from the target viewpoint (rightmost). To visualize tokens, we color-code each source token by its position in the source image, and preserve this color after warping, so the color of each token in the warped views indicates its source location. With pixel-wise backward warping , the target-view image suffers from local pixel distortions caused by depth noise, leading the MLLM to answer incorrectly. In contrast, backward token warping with nearest fetching preserves the semantic content while shifting viewpoint, allowing the MLLM to produce the correct ordering of the toys. (Source and target images were captured manually.)
Figure A4 : Qualitative Sample 3. Given the source image (leftmost), the question asks to describe the red object (red box) placed on the left side of the omelet (blue box) when viewed from the target viewpoint (rightmost). To visualize tokens, we color-code each source token by its position in the source image, and preserve this color after warping, so the color of each token in the warped views indicates its source location. When using pixel-wise forward warping , the warped image exhibits local pixel distortions due to depth prediction noise and holes caused by magnification. Consequently, given this warped RGB image, the MLLM incorrectly answers that the object is “a piece of fruit” . In contrast, with backward token warping and adaptive fetching , the MLLM correctly identifies the object as a “bottle” , more specifically “containing a condiment or sauce” and “ketchup” . This further highlights the advantage of warping in token space rather than pixel space when transferring source content to a target view. (Source and target images are from DL3DV-10K [ 54 ] .)
Figure A5 : Qualitative Sample 4. Given the source image (leftmost), the question asks to describe the object that is located on the right side of the white mug (red box) when viewed from the target viewpoint (rightmost). To visualize tokens, we color-code each source token by its position in the source image, and preserve this color after warping, so the color of each token in the warped views indicates its source location. With pixel-wise forward warping , the warped image shows distorted local details due as the forward warping distributes the source image pixels to a sparse grid in the target image. Consequently, the MLLM fails to accurately describe the bottle on the right side, and instead replies “piece of fabric” and “part of a bag” , which are not visible in the target image. On the other hand, when using backward token warping with adaptive fetching , the MLLM describes the specified object as “a bottle” and “a type of beverage” which is accurate when seen from the target image. These results again show that our proposes backward token warping can provide a robust way of transferring source image information to the target viewpoint. (Source and target images are from BLINK [ 30 ] .)
More in this topic
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,091 papers