Seeking Universal Shot Language Understanding Solutions

Haoxin Liu; Harshavardhan Kamarthi; Zhiyuan Zhao; Hongjie Chen; B. Aditya Prakash

cs.LGMar 19, 2026

Seeking Universal Shot Language Understanding Solutions

Authors: Haoxin Liu, Harshavardhan Kamarthi, Zhiyuan Zhao, Hongjie Chen, B. Aditya Prakash

Organizations: Georgia Institute of Technology · Dolby Laboratories

Abstract

Shot language understanding (SLU) is crucial for cinematic analysis but remains challenging due to its diverse cinematographic dimensions and subjective expert judgment. While vision-language models (VLMs) have shown strong ability in general visual understanding, recent studies reveal judgment discrepancies between VLMs and film experts on SLU tasks. To address this gap, we introduce SLU-SUITE, a comprehensive training and evaluation suite containing 490K human-annotated QA pairs across 33 tasks spanning six film-grounded dimensions. Using SLU-SUITE, we originally observe two insights into VLM-based SLU from: the model side, which diagnoses key bottlenecks of modules; the data side, which quantifies cross-dimensional influences among tasks. These findings motivate our universal SLU solutions from two complementary paradigms: UniShot, a balanced one-for-all generalist trained via dynamic-balanced data mixing, and AgentShots, a prompt-routed expert cluster that maximizes peak dimension performance. Extensive experiments show that our models outperform task-specific ensembles on in-domain tasks and surpass leading commercial VLMs by 22% on out-of-domain tasks.

Figures & tables

Human-Labeled Dataset	Scale	Training Supp.	OOD Eval.
ShotBench Liu et al. ()	8 tasks $\sim$ 70K QA	✓	$\times$
CameraBench Lin et al. ()	5 tasks $\sim$ 150K labels	✓	$\times$
CineTechBench Wang et al. ()	7 tasks $\sim$ 0.7K QA	$\times$	N/A
MovieCuts Pardo et al. (2022)	1 task $\sim$ 110K clips	✓	$\times$
SLU-SUITE (Ours)	33 tasks $\sim$ 490K QA	✓	✓

Table 1: Comparison with representative human-labeled SLU datasets. Scale reports the number of tasks and the approximate data size. Training Supp. indicates whether official splits support fine-tuning/training. OOD Eval. indicates whether the benchmark supports controlled out-of-domain evaluation by design (e.g., unseen tasks), compared with the common in-domain (ID) evaluation. Our SLU-SUITE covers a broader set of tasks, provides large-scale human-labeled data, and explicitly supports both ID and OOD evaluations to advabce general SLU.

High-Level Dimension	Detailed Tasks	Num. Sources	Num. Samples
Lighting	4 tasks: SourceCondition ; Style ; Attribute ; ColorPalette	2	925
Composition	3 tasks: CompositionRule ; CompositionPattern ; VisualWeightPlacement	3	4,841
Viewpoint	6 tasks: AngleBasic ; AngleExtended ; AngleFinegrained ; AngleCartoon ; Height ; HeightCartoon	4	37,742
Coverage	9 tasks: ScaleBasic ; ScaleClassic ; ScaleExtended ; ScaleFinegrained ; ScaleCartoon ; ScaleHistorical ; Staging ; StagingScaleMix ; FocalLength	7	51,693
Motion	10 tasks: MoveCoarse ; MoveCompoundA ; MoveCompoundB ; MoveCaptioning ; VQAComplexity ; VQAMovement ; VQAShaking ; VQASpeed ; VQAPresence ; VQAMixedType	5	282,297
Cuts	1 task: InterShotCutType	1	109,827

Table 2: SLU-SUITE summary by high-level dimension. Each row groups task variants under one dimension. The Detailed Tasks lists task variants in that dimension (count in bold). We define a task variant by a specific option set (label space/taxonomy); thus, semantically similar tasks from different sources are treated as different variants when their option sets differ. Num. Sources denotes the number of contributing datasets, and Num. Samples denotes the number of labeled QA pairs. Overall, SLU-SUITE includes 33 task variants from 11 datasets, totaling 487,325 human-labeled QA pairs and covering 53,569 unique images and 154,451 unique videos after deduplication.

	LoRA Target
Dimension	Base	Connector	Vision	LM	LM+Connector	All
Lighting	0.533	0.533	0.524	0.610	0.661	0.616
Composition	0.062	0.412	0.277	0.546	0.623	0.579
Viewpoint	0.500	0.725	–	0.750	0.780	0.801
Coverage	0.457	0.609	0.627	0.619	0.667	0.633
Motion	0.602	0.647	0.605	0.713	0.724	0.721

Table 3: Results of updating different modules of VLM (Qwen3-VL-8B) for SLU. Each row trains and tests on a single SLU dimension, thereby removing cross-dimensional transfer effects. LM+Connector achieves the best average performance while using about

15\%

fewer trainable parameters than All . Vision-only adaptation on viewpoint resulted in model collapse, i.e., incapable of solving the OOD tasks; see App. E

Figure 1: Cross-dimensional transfer matrix. Each column denotes the source training dimension and each row denotes the target test dimension. The NA column corresponds to initial performance of the VLM backbone (Qwen3-VL-8B). Transfer is broadly useful, but clearly uneven across targets.

	General-purpose VLMs				Existing SLU	Ours		Task-specific Ensemble
12 In-Domain Tasks	Qwen3-VL-8B	Gemini-2.5-Flash	Gemini-3.0-Flash	Gemini-3.0-Pro	ShotVL-7B	UniShot	AgentShots	12 task-specific SFT Qwen3-VL-8B
CompositionRule	0.020	0.005	0.040	0.040	0.775	0.790	0.785	0.784
ScaleExtended	0.145	0.165	0.125	0.170	0.115	0.290	0.255	0.320
ScaleHistorical	0.670	0.770	0.850	0.825	0.740	0.900	0.895	0.855
ScaleCartoon	0.715	0.675	0.650	0.660	0.460	0.940	0.925	0.875
ScaleBasic	0.650	0.465	0.595	0.555	0.425	0.875	0.875	0.865

Table 4: ID results on 12 seen task variants. We compare general-purpose VLMs, an existing SLU model, our two universal solutions, and 12 task-specific Qwen3-VL-8B counterparts. Avg. Acc denotes the unweighted mean across tasks. Count #1 reports the number of tasks on which a method ranks first. UniShot achieves the best average accuracy ( 0.759 ), while AgentShots remains nearly identical (0.758). Both of our universal models outperform the ensemble of 12 task-specific counterparts (0.740), suggesting positive transfer across seen SLU tasks.

	General-purpose VLMs				Existing SLU	Ours
11 Out-of-Domain Tasks	Qwen3-VL-8B	Gemini-2.5-Flash	Gemini-3.0-Flash	Gemini-3.0-Pro	ShotVL-7B	UniShot	AgentShots
CompositionPattern	0.083	0.208	0.158	0.525	0.133	0.433	0.558
ScaleClassic	0.371	0.486	0.600	0.436	0.507	0.414	0.593
FocalLength	0.217	0.333	0.467	0.433	0.300	0.367	0.400
AngleExtended	0.650	0.575	0.825	0.792	0.608	0.608	0.650
MoveCompoundB	0.815	0.750	0.810	0.585	0.805	0.880	0.880

Table 5: OOD results on 11 unseen task variants. OOD tasks remain within the same six high-level shot-language dimensions, but introduce unseen tasks, either new option sets ( ScaleClassic ) or new task types ( FocalLength ). Avg. Acc. denotes the unweighted mean across tasks. Count #1 reports the number of tasks on which a method ranks first. AgentShots delivers the strongest OOD generalization, achieving the best average accuracy ( 0.666 ) and ranking first on 7/11 task variants. UniShot remains competitive at 0.617 and still exceeds all other solutions.

Figure 2: Ablation of data mixing strategies. Accuracy is normalized by AgentShots_naive ; $1.0$ means matching specialist trained only on in-dimension data. Compared with UniShot_naive , UniShot improves the weakest dimensions and average performances, producing a more balanced single-model generalist. Compared with AgentShots_naive , AgentShots improves all six dimensions and achieves the best result on 5 of 6. MoLE Wu et al. () serves as advanced multi-LoRA baseline, testing the performances of end-to-end data mixing. See App. I.1 for raw results.

Figure 3: Results of dataset variants. At a matched 4K budget, heterogeneous (multi-source, i.e., randomly sampled from our SLU-SUITE ) dataset yields a significant improvement over the homogeneous dataset (mainly ShotBench). Further scaling the heterogeneous pool up to 410K provides progressive performance improvement. See detailed results in App. 11 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Unified multiple-choice prompt template used for the majority of SLU-SUITE tasks, including composition, coverage, viewpoint, lighting, motion classification, and cut-type recognition. The multimodal placeholder is always placed at the first line.

Dimension	Incoming + / 5	Mean Incoming Gain	Outgoing + / 5	Mean Outgoing Gain
Lighting	5/5	+0.122	3/5	+0.078
Composition	5/5	+0.299	4/5	+0.044
Viewpoint	5/5	+0.032	5/5	+0.108
Coverage	1/5	-0.010	5/5	+0.115
Motion	5/5	+0.056	4/5	+0.094
Cuts	4/5	+0.008	4/5	+0.069

Appendix

Table 6: Summary of cross-dimensional transfer utility. “Incoming” measures how a target benefits from external source dimensions. “Outgoing” measures how much a source helps other targets. Mean gains are averaged over all five off-diagonal targets relative to the Base model.

Target-side transfer-in			Source-side transfer-out
Target	Positive Count	Mean Gain	Source	Positive Count	Mean Gain
Lighting	5 / 5	+0.122	Lighting	3 / 5	+0.078
Composition	5 / 5	+0.299	Composition	4 / 5	+0.044
Viewpoint	5 / 5	+0.032	Viewpoint	5 / 5	+0.108
Coverage	1 / 5	-0.010	Coverage	5 / 5	+0.115
Motion	5 / 5	+0.056	Motion	4 / 5	+0.094

Appendix

Table 7: Transfer-in and transfer-out summary derived from Fig. 1 . Positive count reports how many off-diagonal transfers improve over the pretrained baseline. Mean gain is computed as the average off-diagonal improvement over the baseline under the natural SLU-SUITE regime.

Dimension	UniShot	UniShot-naive	AgentShots	AgentShots-naive	MoLE
Composition	0.6117	0.6325	0.6717	0.6230	0.6050
Coverage	0.6530	0.6560	0.6718	0.6670	0.6211
Cuts	0.4000	0.3550	0.4250	0.3800	0.3250
Lighting	0.6303	0.5902	0.6833	0.6610	0.7083
Motion	0.7599	0.7309	0.7667	0.7240	0.6899
Viewpoint	0.7767	0.7690	0.7970	0.7800	0.7277

Appendix

Table 8: Recipe ablation (absolute scores): AgentShots is best on average, while UniShot improves balance over naive mixing. UniShot-naive directly mixes all six dimensions without dynamic balancing. AgentShots-naive trains each expert only on its own target-dimension data. MoLE uses the same six experts and top-1 activation budget as AgentShots, but learns routing end-to-end. Dynamic balancing mainly improves the weakest dimensions of UniShot, while target-aware mixtures consistently improve AgentShots over isolated expert training.

Dimension	UniShot	UniShot-naive	AgentShots	AgentShots-naive	MoLE
Lighting	0.9536	0.8929	1.0337	1.0000	1.0716
Composition	0.9819	1.0152	1.0782	1.0000	0.9711
Viewpoint	0.9958	0.9859	1.0218	1.0000	0.9329
Coverage	0.9790	0.9835	1.0072	1.0000	0.9312
Motion	1.0496	1.0095	1.0590	1.0000	0.9529
Cuts	1.0526	0.9342	1.1184	1.0000	0.8553

Appendix

Table 9: Recipe ablation (normalized scores): UniShot improves the floor, while AgentShots improves every dimension. Results are normalized by AgentShots-naive (

=1.0

). Values above

1.0

indicate improvement over the naive routed-specialist baseline. We additionally report the average, worst-case, and second worst-case performance for each method across the six dimensions. For AgentShots-naive , all entries are

1.0

by definition.

Dimension	Raw Qwen3.5	UniShot on SLU-SUITE
Composition	0.0617	0.6100
Coverage	0.4569	0.6662
Cuts	0.2300	0.3350
Lighting	0.5326	0.7076
Motion	0.6018	0.7407
Viewpoint	0.5000	0.8027

Appendix

Table 10: Transfer to a newer backbone: SLU-SUITE supervision remains effective on Qwen3.5-VL-9B. We replace Qwen3-VL-8B with Qwen3.5-VL-9B and keep the UniShot training setting unchanged. SLU-SUITE supervision consistently improves over the raw model across all six shot-language dimensions.

Dimension	Raw	Homogeneous 4K	Heterogeneous 4K	Heterogeneous 100K	Heterogeneous 410K
Composition	0.0617	0.3775	0.5858	0.6217	0.6167
Coverage	0.4569	0.5178	0.5878	0.6573	0.6407
Cuts	0.2300	0.2800	0.3200	0.3550	0.4000
Lighting	0.5326	0.6439	0.6432	0.5939	0.6303
Motion	0.6018	0.6293	0.6882	0.7300	0.7617
Viewpoint	0.5000	0.5057	0.6853	0.7927	0.7753

Appendix

Table 11: Data heterogeneity and scale (full breakdown): heterogeneity gives the main jump, while scale adds uneven gains. Raw is the pretrained model without SLU fine-tuning. Homogeneous 4K and Heterogeneous 4K use the same training size but differ in source diversity. Heterogeneous 100K and Heterogeneous 410K further scale the same heterogeneous pool. Note that the lighting split is the same in the homogeneous and heterogeneous 4K settings, so no clear gain is expected for lighting at that scale.

Panel of Figure 5 — Figure 5: Causal view of subjective supervision in SLU. The source $s$ affects both the visual distribution and the annotation mechanism. We separate semantic evidence $H=\phi(X)$ from source-dependent annotation effects: a taxonomy/readout operator $A_{s}$ and a preference term $b_{s}$ .

Video Understanding

Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.

See all 116 papers

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models
Sep 23, 2026Shuo Xing, Pooja Verlani, Balu Adsumilli +1Video Understanding Video-Language Model Evaluation
Texas A&M University, College Station, USA · Google Inc., Mountain View, USA
How Well Do Vision-Language Models Understand Movies?
Jun 6, 2025Emmanouil Zaranis, António Farinhas, Saul Santos +28Video-Language Model Evaluation Multimodal Understanding
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ILLC, University of Amsterdam +12
Natural Language Camera Movement Understanding
Jul 3, 2026Yuwen Tan, Joey Huang, Jin Huang +2VLM Evaluation Video-Language Models
Boston University, Boston, MA 02215, USA · Pixocial Technology, Bellevue, WA 98004, USA

Sep 23, 2026J/K move · Enter open · S save

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

Shuo Xing, Pooja Verlani, Balu Adsumilli, Zhengzhong Tu

Texas A&M University, College Station, USA · Google Inc., Mountain View, USA

Video Understanding Video-Language Model Evaluation Video-Language Models

Open paper AskPDF arXiv

Cinematography, the craft of visual storytelling through framing, lighting, and camera operation, fundamentally shapes how audiences perceive and emotionally engage with video content. While Large Vision Language Models (LVLMs) have made remarkable progress in video question answering, existing benchmarks primarily focus on identifying low-level techniques rather than understanding their storytelling impact. To address this, we introduce CinematicVQA, the first-of-its-kind benchmark for cinematic video understanding that goes beyond technique recognition to evaluate film-grammar reasoning, utilizing our introduced Cinematic Scene Graph (CSG), a structured representation that links filming techniques to their perceptual effects and narrative functions. Through comprehensive evaluation of state-of-the-art LVLMs, we reveal a striking semantic gap: models consistently perform higher on describing visual presentations than on identifying the underlying techniques. Surprisingly, Chain-of-Thought prompting fails to provide consistent gains and degrades performance for most models, suggesting that current LVLMs lack sufficient cinematic domain knowledge to benefit from step-by-step reasoning. Fine-tuning on CinematicVQA-train yields consistent improvements, particularly for narrative function and multi-hop reasoning. Overall, CinematicVQA serves both as a rigorous benchmark for cinematic evaluation in LVLMs and as a practical dataset for training more film-aware video models.

Abstract

Video Understanding

Explore similar work

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

How Well Do Vision-Language Models Understand Movies?

Natural Language Camera Movement Understanding