BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning
Authors: Zekun Qian, Ruize Han, Wei Feng
Organizations: School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China
Object-level spatial-temporal understanding is essential for video question answering, yet existing multimodal large language models (MLLMs) encode frames holistically and lack explicit mechanisms for fine-grained object grounding. Recent work addresses this by serializing bounding box coordinates as text tokens, but this text-coordinate paradigm suffers from a fundamental modality mismatch: object information is inherently visual, yet encoding it as text incurs a high token cost that forces aggressive temporal downsampling. We propose BoxTuning, which resolves this mismatch by injecting object spatial-temporal information directly into the visual modality. Colored bounding boxes and trajectory trails are rendered onto video frames as visual prompts, with only a concise color-to-object legend retained as text. This reduces the token cost significantly, achieving 87-93% text token reduction in practice. It also preserves full temporal resolution, where the trajectory trails further encode inter-frame motion direction and speed within each keyframe, recovering fine-grained dynamics that text-coordinate methods are forced to discard. Experimental results on five video QA benchmarks (CLEVRER, Perception Test, STAR, NExT-QA, IntentQA) show that BoxTuning surpasses text-coordinate baselines on spatially oriented tasks and nearly eliminates the accuracy degradation observed on reasoning-centric tasks, establishing visual prompting as a more natural and efficient paradigm for conveying object information to video MLLMs.
Figures & tables
Figure 1: Comparison of object-centric representation strategies for video MLLMs. Left : ObjectMLLM [ 31 ] serializes per-frame bounding box coordinates as text tokens ( e.g . , “ (Object 0) adult: frame 0 [73 30 92 100] … ”), producing 1,218 tokens for a single video. The original keyframes lack intermediate dynamics, leading to action misinterpretation. Right : BoxTuning renders colored bounding boxes and trajectory trails directly on the video frames, with only a concise color-mapped text legend ( e.g . , “ (1) Red box: Adult; (2) Blue box: Elephant; … ”, 182 tokens). The trajectory-augmented keyframes capture continuous dynamics, enabling accurate temporal reasoning. Overall, BoxTuning achieves 85% token savings (in this case) while bridging temporal gaps that text-coordinate methods miss.
Figure 2: Overview of the BoxTuning framework. Visually prompted frames, produced by rendering colored bounding boxes and trajectory trails from an off-the-shelf detector (YOLO-World [ 5 ] ) and tracker (SAM 2 [ 29 ] ), are fed into the vision encoder together with a concise text legend mapping colors to object classes and the question. The vision encoder is kept frozen, while the vision projector (STC connector) is fully fine-tuned and the LLM is adapted via LoRA [ 8 ] .
Figure 3: Visual prompt examples on CLEVRER video frames. (a) Bounding boxes alone provide spatial localization of each object. (b) Adding trajectory trails encodes motion direction and speed across intermediate frames, aiding temporal reasoning. The text legend (not shown) maps each color to its object class.
Component
ObjectMLLM
BoxTuning (Ours)
Per-entry cost
∼18 tok (frame + coords)
∼9 tok (index + label + color)
Entries per video
N×T≈8×9=72
N≈8
Total object text tok
∼18×72=1,296
∼9×8= 72
Table 1: Token cost breakdown for a representative NExT-QA video ( N≈8 objects, T≈9 temporally sampled frames per object).
Benchmark
ObjectMLLM
BoxTuning
Reduction
CLEVRER
611
61
90.1%
Perception Test
1,241
120
90.4%
STAR
761
96
87.4%
NExT-QA
1,184
77
93.5%
IntentQA
1,204
85
93.0%
Average
1,000
88
90.8%
Table 2: Average text tokens for object information per sample. ObjectMLLM [ 31 ] serializes bounding box coordinates as text. BoxTuning uses only a concise color legend.
Method
CLEVRER
Perc. Test
STAR
NExT-QA
IntentQA
VideoLLaMA2 (base)
67.9
66.0
66.5
79.8
76.7
+ ObjectMLLM (text coord.)
77.6 +9.7
66.6 +0.6
67.2 +0.7
78.5 -1.3
75.5 -1.2
+ BoxTuning (ours)
78.5 +10.6
67.2 +1.2
67.7 +1.2
79.7 -0.1
76.5 -0.2
Table 3: Accuracy (%) on five video QA benchmarks. All methods use VideoLLaMA2-7B with LoRA fine-tuning. Baseline numbers are from [ 31 ] . Best in bold . Green / red indicate change relative to the base model.
Method
Size
CLEVR.
Perc.T.
STAR
NExT
Intent
LLaVA-NeXT-Vid. [ 48 ]
7B
38.4 ∗†
49.3 ∗
–
–
–
VideoLLaMA2 [ 6 ]
7B
45.6 ∗†
51.4 ∗
57.1 ∗†
74.1 ∗†
73.8 ∗†
SeViLA [ 41 ]
3B
–
62.0
64.9
73.8
–
ViLA [ 35 ]
3B
–
–
67.1
75.6
–
ObjectMLLM [ 31 ]
7B
77.6
66.6
67.2
78.5
75.5
BoxTuning (ours)
7B
78.5
67.2
67.7
79.7
76.5
Table 4: Comparison with existing video MLLMs. Accuracy (%) is reported.
Variant
CLEVRER
Perc. Test
STAR
NExT-QA
IntentQA
Box only
77.0
66.3
66.8
79.3
76.1
Box + Legend
77.3
66.5
67.2
79.4
76.2
Box + Legend + Trail
78.5
67.2
67.7
79.7
76.5
Table 5: Component-wise ablation. Accuracy (%) is reported.
Trail length L
10
20
30
40
60
Adaptive
CLEVRER
77.6
78.0
78.2
78.1
78.0
78.5
Perc. Test
66.5
66.9
67.0
67.0
66.9
67.2
Table 6: Ablation study on trajectory trail length L . Accuracy (%) is reported.
Width
1
2
3
5
8
Box
77.8
78.2
78.5
78.0
77.1
Trail
78.0
78.5
78.3
77.4
76.8
Table 7: Ablation study on rendering line width on CLEVRER using accuracy (%).