BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning
Authors: Zekun Qian, Ruize Han, Wei Feng
Organizations: School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China
Object-level spatial-temporal understanding is essential for video question answering, yet existing multimodal large language models (MLLMs) encode frames holistically and lack explicit mechanisms for fine-grained object grounding. Recent work addresses this by serializing bounding box coordinates as text tokens, but this text-coordinate paradigm suffers from a fundamental modality mismatch: object information is inherently visual, yet encoding it as text incurs a high token cost that forces aggressive temporal downsampling. We propose BoxTuning, which resolves this mismatch by injecting object spatial-temporal information directly into the visual modality. Colored bounding boxes and trajectory trails are rendered onto video frames as visual prompts, with only a concise color-to-object legend retained as text. This reduces the token cost significantly, achieving 87-93% text token reduction in practice. It also preserves full temporal resolution, where the trajectory trails further encode inter-frame motion direction and speed within each keyframe, recovering fine-grained dynamics that text-coordinate methods are forced to discard. Experimental results on five video QA benchmarks (CLEVRER, Perception Test, STAR, NExT-QA, IntentQA) show that BoxTuning surpasses text-coordinate baselines on spatially oriented tasks and nearly eliminates the accuracy degradation observed on reasoning-centric tasks, establishing visual prompting as a more natural and efficient paradigm for conveying object information to video MLLMs.
Figures & tables
Figure 1: Comparison of object-centric representation strategies for video MLLMs. Left : ObjectMLLM [ 31 ] serializes per-frame bounding box coordinates as text tokens ( e.g . , “ (Object 0) adult: frame 0 [73 30 92 100] … ”), producing 1,218 tokens for a single video. The original keyframes lack intermediate dynamics, leading to action misinterpretation. Right : BoxTuning renders colored bounding boxes and trajectory trails directly on the video frames, with only a concise color-mapped text legend ( e.g . , “ (1) Red box: Adult; (2) Blue box: Elephant; … ”, 182 tokens). The trajectory-augmented keyframes capture continuous dynamics, enabling accurate temporal reasoning. Overall, BoxTuning achieves 85% token savings (in this case) while bridging temporal gaps that text-coordinate methods miss.
Figure 2: Overview of the BoxTuning framework. Visually prompted frames, produced by rendering colored bounding boxes and trajectory trails from an off-the-shelf detector (YOLO-World [ 5 ] ) and tracker (SAM 2 [ 29 ] ), are fed into the vision encoder together with a concise text legend mapping colors to object classes and the question. The vision encoder is kept frozen, while the vision projector (STC connector) is fully fine-tuned and the LLM is adapted via LoRA [ 8 ] .
Figure 3: Visual prompt examples on CLEVRER video frames. (a) Bounding boxes alone provide spatial localization of each object. (b) Adding trajectory trails encodes motion direction and speed across intermediate frames, aiding temporal reasoning. The text legend (not shown) maps each color to its object class.
Component
ObjectMLLM
BoxTuning (Ours)
Per-entry cost
∼18 tok (frame + coords)
∼9 tok (index + label + color)
Entries per video
N×T≈8×9=72
N≈8
Total object text tok
∼18×72=1,296
∼9×8= 72
Table 1: Token cost breakdown for a representative NExT-QA video ( N≈8 objects, T≈9 temporally sampled frames per object).
Benchmark
ObjectMLLM
BoxTuning
Reduction
CLEVRER
611
61
90.1%
Perception Test
1,241
120
90.4%
STAR
761
96
87.4%
NExT-QA
1,184
77
93.5%
IntentQA
1,204
85
93.0%
Average
1,000
88
90.8%
Table 2: Average text tokens for object information per sample. ObjectMLLM [ 31 ] serializes bounding box coordinates as text. BoxTuning uses only a concise color legend.
Method
CLEVRER
Perc. Test
STAR
NExT-QA
IntentQA
VideoLLaMA2 (base)
67.9
66.0
66.5
79.8
76.7
+ ObjectMLLM (text coord.)
77.6 +9.7
66.6 +0.6
67.2 +0.7
78.5 -1.3
75.5 -1.2
+ BoxTuning (ours)
78.5 +10.6
67.2 +1.2
67.7 +1.2
79.7 -0.1
76.5 -0.2
Table 3: Accuracy (%) on five video QA benchmarks. All methods use VideoLLaMA2-7B with LoRA fine-tuning. Baseline numbers are from [ 31 ] . Best in bold . Green / red indicate change relative to the base model.
Method
Size
CLEVR.
Perc.T.
STAR
NExT
Intent
LLaVA-NeXT-Vid. [ 48 ]
7B
38.4 ∗†
49.3 ∗
–
–
–
VideoLLaMA2 [ 6 ]
7B
45.6 ∗†
51.4 ∗
57.1 ∗†
74.1 ∗†
73.8 ∗†
SeViLA [ 41 ]
3B
–
62.0
64.9
73.8
–
ViLA [ 35 ]
3B
–
–
67.1
75.6
–
ObjectMLLM [ 31 ]
7B
77.6
66.6
67.2
78.5
75.5
BoxTuning (ours)
7B
78.5
67.2
67.7
79.7
76.5
Table 4: Comparison with existing video MLLMs. Accuracy (%) is reported.
Variant
CLEVRER
Perc. Test
STAR
NExT-QA
IntentQA
Box only
77.0
66.3
66.8
79.3
76.1
Box + Legend
77.3
66.5
67.2
79.4
76.2
Box + Legend + Trail
78.5
67.2
67.7
79.7
76.5
Table 5: Component-wise ablation. Accuracy (%) is reported.
Trail length L
10
20
30
40
60
Adaptive
CLEVRER
77.6
78.0
78.2
78.1
78.0
78.5
Perc. Test
66.5
66.9
67.0
67.0
66.9
67.2
Table 6: Ablation study on trajectory trail length L . Accuracy (%) is reported.
Width
1
2
3
5
8
Box
77.8
78.2
78.5
78.0
77.1
Trail
78.0
78.5
78.3
77.4
76.8
Table 7: Ablation study on rendering line width on CLEVRER using accuracy (%).
Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.
Despite remarkable progress in multimodal understanding, current MLLMs still exhibit limitations in video text understanding, particularly when semantics emerge through the integration of temporally distributed textual cues across multiple frames. This perception challenge fundamentally differs from static image text understanding, yet existing datasets fail to capture: the vast majority of questions remain answerable from single frames, inadequately reflecting real-world video text comprehension demands. To address this, we present ViTexQA, a large-scale video-text QA dataset, and FrameThinker for robust multi-frame temporal reasoning. We build ViTexQA via a quality-controlled Chain-of-Thought (CoT) annotation pipeline boosted with temporal constraints; all its QA pairs demand cross-frame text fusion to solve, enforcing true temporal reliance. FrameThinker adopts two-stage training for explicit temporal modeling: CoT-Guided Supervised Fine-Tuning (SFT) generates frame-aware reasoning chains, followed by Temporally-grounded Reinforcement Learning (RL) optimized with multi-frame coherence rewards. Evaluations show our method outperforms SOTA baselines on ViTexQA, lifting ROUGE-L by 6.3%.
Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the referring expression without any temporal reasoning. To address this, we introduce novel motion-centric probing, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the ordering of motion. Consequently, we introduce MoCentric-Bench, a motion-centric benchmark designed to evaluate video MLLMs based on their ability to capture the interaction between motion and language, rather than relying primarily on static cues. We further establish strong single-image baselines that either match or surpass prior methods. Finally, we explore a simple motion-centric adaptation that provides state-of-the-art performance on our MoCentric-Bench. Code is available at https://msiam.github.io/PixFoundationSeries/.