Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering

Anas Filali Razzouki; Killian Steunou; Khalil Guetari; Thomas Kling; Mounîm El-Yacoubi; Yannis Tevissen

cs.CVOct 7, 2026

Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering

Authors: Anas Filali Razzouki, Killian Steunou, Khalil Guetari, Thomas Kling, Mounîm El-Yacoubi, Yannis Tevissen

Organizations: Télécom SudParis, Institut Polytechnique de Paris, France · Moments Lab Research, France

Abstract

Linking people's appearance and actions to character identities is essential for understanding video narratives. We present a framework for identity-aware video captioning and person-centric question answering that combines automatic character identification, explicit spatial grounding, and task-specific adaptation. Starting from LSMDC v2 movie clips, our pipeline matches detected faces to actor reference images, tracks characters across frames, and builds inputs with identity-linked bounding boxes. A strong vision-language model generates identity-aware captions and questions, which are manually verified and filtered to create a benchmark of 750 captioned clips and 3,000 person-centric questions. We study five grounding strategies combining textual coordinates with visual face or estimated person boxes across Video-MLLM families at roughly 2B, 4B, and 8B parameters and larger frontier models. Combining visual face boxes with textual coordinates yields the most consistent performance across scales and significantly improves overall performance over coordinates alone. Smaller models tend to over-assign known identities when the queried person is not grounded, while larger models better recognize such UNIDENTIFIED cases. We introduce BAC by LoRA fine-tuning Qwen models at 2B, 4B, and 8B scales on about 32K identity-aware captioned clips. Across all scales, BAC outperforms every other evaluated model family of comparable size. BAC-8B reaches 93.20% overall QA accuracy, ranking behind only GPT-5.6 Sol among the frontier models evaluated in our study. Overall, explicitly communicating who is where, together with lightweight task-specific adaptation, substantially improves identity-aware video understanding without changing the underlying architecture. We release the benchmark, training data, code, and BAC checkpoints at https://github.com/momentslab/beyond-anonymous-captions.

Figures & tables

Figure 1: Example of identity-aware captioning and person-centric QA using character identities grounded across sampled video frames.

		Captioning			Person-Centric QA
Size	Grounding	Overall $\uparrow$	Identity $\uparrow$	Halluc. $\downarrow$	Overall $\uparrow$	Named $\uparrow$	UNID $\uparrow$
2B	FTC	5.74	47.47	7.60	53.87	75.60	5.18
	FVB	5.92	36.80	10.27	68.27	86.55	27.32
	PVB	5.52	27.54	8.28	66.13	80.71	33.48
	FVB+FTC	7.61	76.53	3.07	68.90	92.48	16.09
	PVB+PTC	7.29	69.87	2.67	67.13	91.27	13.07

Table 1: Comparison of identity-grounding strategies across Qwen3-VL model sizes. Captioning is evaluated using overall quality, identity accuracy, and hallucination rate. QA performance is reported over all questions, separately for Named and UNIDENTIFIED (UNID) identities.

Face-area bin	N	FVB+FTC	PVB+PTC	$\Delta$ (P–F)	$p$
Small $<4{,}000$ px 2	85	80.0%	91.8%	+11.8	0.021
Medium $4{,}000$ – $269{,}999$ px 2	1,865	92.8%	91.3%	-1.5	0.009
Large $\geq 270{,}000$ px 2	124	96.8%	91.1%	-5.6	0.016

Table 2: QA accuracy of FVB+FTC and PVB+PTC grounding methods with Qwen3-2B, stratified by face bounding-box area.

\Delta

denotes the accuracy difference between PVB+PTC and FVB+FTC.

Panel of Figure 2 — Figure 2: Qualitative identity-aware QA examples. Identity-mapping colors match the bounding-box colors in each frame. (a) BAC correctly predicts UNIDENTIFIED ; (b) BAC correctly identifies the queried named person; (c) BAC fails to identify the queried named person, while only GPT-5.6 Sol succeeds. ∗ : GPT-5.6 Sol with thinking.

		Captioning			Person-Centric QA
Size	Model	Overall $\uparrow$	Identity $\uparrow$	Halluc. $\downarrow$	Overall $\uparrow$	Named $\uparrow$	UNID $\uparrow$
2B	BAC (ours)	8.09	85.33	3.20	80.33	95.90	45.46
	Qwen3	7.61	76.53	3.07	68.90	92.48	16.09
	InternVL3.5	5.59	24.86	4.48	63.90	91.18	2.81
4B	BAC (ours)	8.48	92.40	0.80	90.13	96.38	76.13
	Qwen3	8.32	88.80	1.07	79.73	93.11	49.78

Table 3: Benchmark comparison across model families and sizes with the FVB+FTC grounding method. Captioning is evaluated using overall quality, identity accuracy, and hallucination rate. QA performance is reported over all questions, separately for Named and UNIDENTIFIED (UNID).

Model	Overall	Named	UNID	Mean	Clip all-correct	Open-weight	Cost ($/1M in/out)
GPT-5.6 Sol (think)	97.07	97.06	97.08	97.07	90.13	No	4.00 / 20.00
GPT-5.6 Sol (no think)	96.30	96.14	96.65	96.40	87.60	No	4.00 / 20.00
BAC-8B (ours)	93.20	93.88	91.68	92.78	76.93	Yes	Self-hosted
BAC-4B (ours)	90.13	96.38	76.13	86.26	66.40	Yes	Self-hosted
Qwen3-VL-235B-A22B	86.10	80.67	98.27	89.47	60.93	Yes	0.287 / 1.147
Gemini 2.5 Flash	82.63	75.80	97.95	86.87	51.60	No	0.30 / 2.50

Table 4: Person-centric QA performance and model accessibility. Mean denotes the average of Named and UNIDENTIFIED (UNID) accuracy. API costs are reported in USD per million input/output tokens; All models use the FVB+FTC grounding method.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Qualitative comparison of face- and person-level grounding for a small-face example. The target character P4 ( Adrianne Palicki ) has a very small face area of 2,222 px 2 and a face height of 52 px. PVB+PTC grounding method correctly identifies Adrianne Palicki across all model sizes, whereas FVB+FTC grounding method predicts Lucas Black at 2B and 4B and UNIDENTIFIED at 8B.

Panel of Figure 4 — Figure 4: Qualitative comparison of face- and person-level grounding for an action question. The target character P1 ( Joaquin ) has a very large face area of 391,897 px 2 and a face height of 760.5 px. FVB+FTC grounding method correctly identifies Joaquin across all three model sizes. PVB+PTC grounding method predicts Abigail at 2B, but correctly identifies Joaquin at 4B and 8B.

Panel of Figure 5 — Figure 5: Training dynamics of the BAC models at three scales. For each model, we report the training loss, evaluation loss, and learning-rate schedule during LoRA fine-tuning. The dashed vertical line indicates the checkpoint with the best evaluation loss.

Panel of Figure 6 — Figure 6: Person-centric QA accuracy across question categories. Top: comparison with frontier models, with categories ordered by GPT-5.6 Sol* performance. Bottom: comparison between BAC-8B and other 8–9B models, with categories ordered by BAC-8B performance. GPT-5.6 Sol* denotes GPT-5.6 Sol with thinking enabled.

Explore similar work

Jul 13, 2026cs.CV

Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

Can a Video Large Language Model (Video-LLM) follow one person through a long video, keeping track of who they are well enough to report, in order, how their outfit changes across a full TV episode? Benchmarks increasingly score this kind of task, and the strongest open-source 7--8B models now reach 37--38% on InfiniBench's global appearance task, which asks exactly that. But does that score come from tracking the named character, or from something easier? We test this with a nine-condition diagnostic protocol applied to three architecturally distinct open-source Video-LLMs, with Gemini~~2.5~~Flash as a frontier reference, and find the accuracy does not come from character tracking. When we change the character named in the question to a different cast member, leaving the video and answer options untouched, the models change their answer only 4--31% of the time, so they are largely ignoring who the question asks about. Breaking that test down by the gender of the swapped name shows why: the models react more when the name is changed to a different-gender character than to a same-gender one (a 13--28 point gap), picking up coarse gender cues but unable to tell same-gender individuals apart. This shallow processing surfaces again when we drop the multiple-choice options and ask the same questions open-endedly: open-source accuracy drops 18--25 points, with none of 151 answers fully correct, versus a 12-point drop for Gemini. Further checks rule out the obvious innocent explanations, adding subtitles, using the most informative frames, or doubling the number of frames all leave character tracking unimproved, so the bottleneck is not how much video the model sees but how it ties that video to the person the question names. We release a diagnostic toolkit for auditing what such benchmark scores actually measure.

Mohammad Al-Ratrout, Shayla Sharmin, Aditya Raikwar +1

Appearance Gender

Ask PDF

Aug 7, 2026cs.CV

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

Real-world video reasoning often involves multimodal, multi-source inputs, whereas existing video reasoning tasks typically assume a simplified video-text setting, limiting identity matching and person-centric reasoning. To bridge this gap, we introduce the Identity-conditioned Queries (ICQ) task, in which models are required to jointly associate and interpret an input video and a reference image of a person, and leverage this conditioning to address identity grounding, behavior understanding, and temporal reasoning, among other challenges. Building on ICQ, we present ISYV (I Seek You in Videos), a systematic solution comprising three components: (1) ISYV-Bench, a challenging evaluation benchmark with 1,377 real-world complex videos and 1,377 question-answer pairs, organized into six difficulty levels spanning capabilities from identity recognition to causal reasoning; (2) ISYV-75K, a large-scale training set of 75K high-quality samples constructed via automated annotation, multi-stage verification, and manual review; and (3) ISYV-Framework, containing an ICQ-oriented model and training strategy for learning to exploit informative video shots without additional shot-level annotations. Extensive experiments show that both mainstream closed-source and open-source MLLMs struggle on ISYV-Bench, especially in cross-domain identity matching and long-horizon tracking. ISYV-Model outperforms strong baselines and in some aspects approaches closed-source performance. Overall, ISYV provides a unified task definition, scalable datasets/benchmarks, and modeling insights for person-centric video reasoning.

Shibo Gao, Chongxiao Wang, Chenglong Huang +10

School of Computing and Communications Lancaster University Lancaster, United Kingdom

Long Videos

Ask PDF

Sep 9, 2026cs.CV

Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.

Zizhen Wang, Bo Feng, Zhengfeng Lai +5

Apple

Captions Long Video Question Answering Multiple-Choice Questions

Ask PDF