Suppressing the Shortcut Hidden Behind the Gains of VideoLLM Fine-Tuning
Organizations: Korea Advanced Institute of Science and Technology (KAIST) · PYLER
Abstract
Supervised fine-tuning (SFT) is the stage in which Video Large Language Models (VideoLLMs) advance their capability to reason over video. However, the VideoLLM also learns to answer from textual regularities of its training data, a shortcut that bypasses the video. The regularities behind the shortcut are hard to define at the data level, and the shortcut forms regardless of training epoch, data size, or backbone. The shortcut is hidden behind a rise in accuracy within the training distribution, yet it is not required for the rise and even costs SFT the gain the video offers under distribution shift. We therefore propose Video-Evidence Training via Orthogonal projection (VETO), which intervenes on the fine-tuning update where the shortcut is written rather than on the data. VETO calibrates the subspace that the text alone occupies in the activations of the base model and projects the update onto its orthogonal complement. The VideoLLM is thus left to ground its answer in the full input, with the shortcut suppressed as it forms and never defined. Across seven VideoLLMs, VETO preserves the gain of SFT on the training distribution and suppresses the shortcut. On six benchmarks under shift it gains more than SFT and answers from the full input rather than the shortcut. Code is available at https://github.com/Youngwoo-git/VETO.
Figures & tables
| Full | Text-only | ||
|---|---|---|---|
| SSv2 | Base | 49.1% | 19.3% |
| SFT | 89.6% | 49.8% | |
| Perception Test | Base | 61.1% | 21.3% |
| SFT | 61.1% | 38.3% |
| Physics | Semantics | Abstraction | Memory | All | ||||||
| Acc. | Text | Acc. | Text | Acc. | Text | Acc. | Text | Acc. | Text | |
| Base | 56.9 | – | 78.0 | – | 52.4 | – | 41.5 | – | 61.1 | – |
| SFT | 59.0 2.0 | +17.4 1.8 | 75.0 1.0 | +17.7 2.9 | 52.2 0.6 | +17.5 2.0 | 45.0 1.5 | +6.2 4.2 | 61.1 0.7 | +17.0 1.1 |
| VETO | 63.0 1.1 | +6.7 3.8 | 78.5 0.6 | +4.0 1.6 | 54.3 1.2 | +3.9 0.7 | 45.0 0.7 | +1.2 2.8 | 64.3 0.7 | +4.9 2.2 |
| Vinoground | AoTBench | MVBench | TVBench | TempCompass | ||||||
| Acc. | Text | Acc. | Text | Acc. | Text | Acc. | Text | Acc. | Text | |
| Base | 55.9 | – | 41.3 | – | 58.0 | – | 42.4 | – | 64.0 | – |
| SFT | 63.5 0.4 | +13.5 1.7 | 53.1 0.2 | +10.9 1.3 | 56.5 0.8 | +7.8 0.5 | 44.3 0.8 | +7.6 0.4 | 65.8 0.6 | +4.0 0.6 |
| VETO | 66.9 2.0 | +7.7 0.8 | 55.2 1.3 | +5.6 2.5 | 58.0 1.0 | +3.5 0.3 | 46.4 0.4 | +3.0 1.5 | 67.6 0.3 | +2.4 0.7 |
| Text-only | Full | Video | ||
|---|---|---|---|---|
| SSv2 | SFT | 16% | 2% | 14% |
| VETO | 68% | 2% | 66% | |
| Perception Test | SFT | 33% | 14% | 19% |
| VETO | 81% | 18% | 63% |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Templ. | Train | Eval | Templates |
|---|---|---|---|---|
| Horizontal | 4 | 8,562 | 756 | Pushing [X] from left to right / Pushing [X] from right to left / Pulling [X] from right to left / Pulling [X] from left to right |
| Vertical | 4 | 7,786 | 789 | Moving [X] up / Moving [X] down / Lifting [X] up completely without letting it drop down / Lifting [X] up completely, then letting it drop down |
| Relative motion | 4 | 3,507 | 483 | Moving [X] and [X] away from each other / Moving [X] and [X] closer to each other / Moving [X] and [X] so they pass each other / Moving [X] and [X] so they collide with each other |
| Camera, horizontal | 4 | 3,495 | 419 | Moving away from [X] with your camera / Turning the camera left while filming [X] / Turning the camera right while filming [X] / Approaching [X] with your camera |
| Camera, vertical | 4 | 3,136 | 266 | Moving [X] away from the camera / Turning the camera downwards while filming [X] / Turning the camera upwards while filming [X] / Moving [X] toward the camera |
| Object-relative | 2 | 1,753 | 383 | Moving [X] away from [X] / Moving [X] closer to [X] |
| Draw | Coverage | Overlap between seeds | Rank | |
|---|---|---|---|---|
| random | 10 | 0.63 | 103 | |
| random | 50 | 0.78 | 171 | |
| random | 100 | 0.83 | 195 | |
| random | 200 | 0.88 | 217 | |
| random | 1,000 | 0.95 | 246 | |
| random | 2,000 | 0.97 | 251 |
| Rank | Coverage | ||
|---|---|---|---|
| 0.5 | 6 | 0.2% | 0.544 |
| 0.7 | 20 | 0.6% | 0.719 |
| 0.8 | 39 | 1.1% | 0.810 |
| 0.9 | 106 | 3.0% | 0.903 |
| 0.95 | 250 | 7.0% | 0.948 |
| 0.99 | 928 | 25.9% | 0.985 |
| VideoLLM | Language model | Params | Full | Text-only |
|---|---|---|---|---|
| Qwen2.5-VL-3B | Qwen2.5 | 3B | 38.5 | 22.4 |
| Qwen2.5-VL-7B | Qwen2.5 | 7B | 49.1 | 19.3 |
| Qwen2.5-VL-32B | Qwen2.5 | 32B | 53.1 | 24.9 |
| Qwen3-VL-8B | Qwen3 | 8B | 51.5 | 30.0 |
| LLaVA-OneVision-7B | Qwen2 | 7B | 39.2 | 24.8 |
| Molmo2-O-7B | Olmo | 7B | 53.5 | 22.6 |
| LoRA rank / scaling / dropout | 16 / 32 / 0.05 |
|---|---|
| Adapted modules | q, k, v, o, gate, up |
| Optimizer | AdamW, lr , weight decay 0.01 |
| Schedule | cosine, 10% warm-up, clip 1.0 |
| Items per update | 8 |
| Precision | bf16 |
| Epochs / patience | up to 10 / 3 on the selection set |
| Full | VCD | |
|---|---|---|
| SFT | 89.6 0.4 | 89.7 0.4 |
| VETO | 89.2 0.2 | 89.2 0.2 |
| Method | One-off (GPU-h) | Per update (s) | One epoch (GPU-h) | SFT |
|---|---|---|---|---|
| SFT | – | 2.24 0.01 | 4.40 0.02 | 1.00 |
| VETO | 0.03 0.00 | 2.36 0.02 | 4.67 0.05 | 1.06 |
| LMH | 17.77 0.33 | 2.26 0.03 | 22.21 0.34 | 5.05 |
| mDPO | 5.28 0.38 | 4.44 0.01 | 13.98 0.39 | 3.18 |
| Accuracy | Text-only accuracy | ||
| Base | – | 49.1 | 19.3 |
| SFT | 3 | 89.6 0.4 | 49.8 0.8 |
| Reshuffled options | 3 | 89.4 0.2 | 49.4 1.0 |
| LMH | 1 | 83.4 | 32.6 |
| mDPO | 1 | 87.0 | 45.3 |
| Random subspace | 3 | 89.6 0.0 | 49.6 2.0 |
| Input | Mean | q | k | v | o | gate | up |
|---|---|---|---|---|---|---|---|
| Text-only, calibration items | 0.950 | 0.950 | 0.950 | 0.950 | 0.950 | 0.950 | 0.950 |
| Text-only, held-out SSv2 | 0.947 | 0.948 | 0.948 | 0.948 | 0.945 | 0.946 | 0.946 |
| Text-only, Perception Test | 0.806 | 0.818 | 0.818 | 0.818 | 0.789 | 0.797 | 0.797 |
| Text-only, Vinoground | 0.823 | 0.832 | 0.832 | 0.832 | 0.807 | 0.818 | 0.818 |
| Text-only, AoTBench | 0.778 | 0.791 | 0.791 | 0.791 | 0.756 | 0.771 | 0.771 |
| Text-only, MVBench | 0.808 | 0.820 | 0.820 | 0.820 | 0.790 | 0.800 | 0.800 |
| SSv2 | Perception Test | |||
|---|---|---|---|---|
| Text-only | Full | Text-only | Full | |
| Maximum ( or ) | 1.39 | 1.39 | 1.10 | 1.10 |
| Base | 1.04 | 0.84 | 0.90 | 0.63 |
| SFT | 0.22 0.08 | 0.03 0.01 | 0.37 0.09 | 0.16 0.05 |
| VETO | 0.93 0.08 | 0.03 0.02 | 0.89 0.02 | 0.20 0.07 |
| Input | Model | ||||
|---|---|---|---|---|---|
| Text-only | Base | 0.26 | 0.27 | 0.23 | 0.20 |
| SFT | 0.26 | 0.26 | 0.40 | 0.50 | |
| VETO | 0.27 | 0.27 | 0.27 | 0.25 | |
| Original video | Base | 0.25 | 0.26 | 0.29 | 0.49 |
| SFT | 0.25 | 0.26 | 0.52 | 0.89 | |
| VETO | 0.27 | 0.26 | 0.47 | 0.89 |
| SSv2 | Perception Test | ||||||
| Full | Blocked | Text-only | Full | Blocked | Text-only | ||
| Base | 49.1 | 19.5 | 19.3 | 61.1 | 21.5 | 21.3 | |
| SFT | 89.6 0.4 | 49.5 1.1 | 49.8 0.8 | 61.1 0.7 | 38.4 0.9 | 38.3 1.1 | |
| Random subspace | 89.6 0.0 | 49.8 1.8 | 49.6 2.0 | 62.1 0.9 | 35.8 0.5 | 35.2 0.6 | |
| VETO | 89.2 0.2 | 30.2 1.2 | 25.3 0.6 | 64.3 0.7 | 25.9 3.1 | 26.2 2.2 | |
| Share of correct answers kept under the block | |||||||
| Dataset | Model | Acc. | Text |
|---|---|---|---|
| NExT-QA | Base | 70.9 | – |
| SFT | 71.7 0.4 | +8.1 0.7 | |
| VETO | 71.6 0.2 | +1.3 0.2 | |
| STAR | Base | 67.9 | – |
| SFT | 70.5 0.2 | +11.6 1.1 | |
| VETO | 70.9 0.6 | +1.3 0.4 |
Video-Language Models
Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.
See all 177 papers