THG-Omni: Traceable Human-Centric Graph Supervision for Video Reasoning
Authors: Jisheng Dang, Kai Wei, Ronghao Lin, Bimei Wang, Hong Peng, Wei-Shi Zheng, Bin Hu, Qi Tian, Tat-Seng Chua
Organizations: School of Information Science & Engineering, Lanzhou University, Lanzhou 730000, China. · the School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China · the School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China · Cloud and AI BU, Huawei, Shenzhen, Guangdong 518129, China · the School of Computing, National University of Singapore, Singapore 119077
Omni-modal models can answer questions about video, but their predictions rarely show which participant, event, or interval supports an answer. Moreover, answer-level supervision alone does not teach a model to interpret structured video evidence. To systematically mitigate this issue, we present THG-Omni, a human-centric graph-supervision framework that links Segment Anything Model~3 participant tracks with temporally grounded Qwen3-VL annotations. Candidate semantic claims are checked against the source video and available audio before they are used to construct training questions, while track geometry provides explicit spatial and temporal supervision. The resulting questions teach an omni-modal reasoning model to interpret participant attributes, actions, affect, interactions, and their timing from a structured graph. At inference, THG-Omni uses a predicted graph constructed from the evaluation video. Experiments on IntentBench show consistent improvements over the unadapted backbone across social reasoning, emotion understanding, and deception perception. On NExT-QA, THG-Omni achieves 89.54% accuracy with the matched graph, compared with 85.10% without graph input. Further validation experiments show that compact graph input achieves 78.22% accuracy, compared with 78.02% using the full graph and 77.58% without graph input. A complementary diagnostic demonstrates that the predicted graph preserves evidence useful for answering questions even without the source video and audio. Together, these findings show that THG-Omni improves human-centric video reasoning while grounding its evidence in identifiable participants and events. Our codes are available at https://anonymous.4open.science/r/review-n94hhfcwp5666/
Figures & tables
Fig. 1: Motivation for traceable graph supervision. This conceptual example contrasts answer-level supervision with identity- and time-grounded graph supervision. The depicted participants and event illustrate the learning objective rather than a benchmark prediction.
Fig. 2: Overview of the proposed traceable human-centric graph-supervision framework. (a) Temporally ordered video evidence. (b) SAM3 grounds and tracks participants, while Qwen3-VL annotates visible attributes, emotions, actions, interactions, and their temporal intervals. (c) The grounded evidence forms a video graph with identity-bound nodes, temporal states, and participant relations. (d) HumanOmniV2 verifies each candidate claim using the source video and available audio; only supported claims are retained. (e) The verified graph supports training of THG-Omni, which is evaluated with video-matched, shuffled, and omitted graph input.
1:
Initialize Dgraph←∅ .
2:
for each training video V and grounding question Q do
3:
Obtain tracks T←gSAM3(V,p(Q)) .
4:
Construct GT=(VT,ET) from track statistics and spatial relations.
5:
Generate semantic claims:
S←gann(V,contact(V),H(T),LC) .
6:
Review identity binding, temporal extent, label validity, and visual support.
Table 3
Fig. 3: IntentBench results and MER24 graph diagnostic. (a) Qwen2.5-Omni-7B and video-matched THG-Omni are compared on 2,688 questions. (b) On 150 MER24 samples without source video or audio, graph input improves exact set match over text-only input.
Social-IQ
EMER
MDPE
Overall
Model / input
Accuracy
Sample F1
Exact
Accuracy
Score
LLaVA-NeXT-Video [ 50 ]
31.58
29.68
12.03
39.20
32.05
HumanOmniV2 [ 7 ]
51.57
80.03
54.89
53.77
53.14
Qwen2.5-Omni-7B [ 51 ]
53.99
68.09
41.35
61.81
55.27
Graph-SFT (w/o graph)
49.66
70.52
40.60
60.30
51.48
Graph-SFT (compact graph)
56.66
71.32
46.62
61.31
57.73
TABLE I: IntentBench results on 2,688 questions. Overall is the sample-weighted aggregate of Social-IQ accuracy, EMER sample F1, and MDPE accuracy. LLaVA-NeXT-Video uses video without audio. Graph-SFT uses a distinct adapter; the THG-Omni rows report different graph-input conditions for the same trained model.
Method
Why (698)
How (632)
What (480)
When (14)
Who/Which (25)
Other (507)
Overall
Qwen2.5-Omni-7B [ 51 ]
53.44
54.11
53.54
57.14
64.00
54.44
53.99
Graph-SFT (w/o graph)
47.99
49.05
50.00
50.00
56.00
52.07
49.66
Graph-SFT (compact graph)
55.01
56.01
57.08
57.14
64.00
58.97
56.66
THG-Omni (w/o graph)
66.19
65.66
66.88
64.29
72.00
71.01
67.28
THG-Omni (shuffled graph)
66.48
65.19
65.83
64.29
72.00
71.40
67.11
THG-Omni (matched graph)
69.63
69.78
70.83
71.43
80.00
74.95
71.18
TABLE II: Social-IQ accuracy (%) across six question types on 2,356 questions.
Fig. 4: Illustrative Social-IQ video-to-graph example. The video frames show two participants and their interaction during a card game. SAM3 provides person tracks and geometric evidence, while semantic annotations describe the observed interaction. The example illustrates participant grounding and temporally localized visual evidence associated with the Social-IQ question. The reference answer is B.
Method
Precision
Recall
Sample F1
Exact
Qwen2.5-Omni-7B [ 51 ]
74.00
68.53
68.09
41.35
Graph-SFT (w/o graph)
76.20
70.10
70.52
40.60
Graph-SFT (compact graph)
78.10
70.80
71.32
46.62
THG-Omni (w/o graph)
89.54
83.97
84.63
66.17
THG-Omni (shuffled graph)
87.97
79.52
81.29
61.65
THG-Omni (matched graph)
93.36
83.91
86.26
66.17
TABLE III: EMER multi-label performance on 133 examples (%).
Fig. 5: Illustrative Social-IQ video-to-graph example. The video frames show the visible participants and the woman’s later reaction. The example illustrates participant grounding and temporally localized visual evidence associated with the Social-IQ question. The reference answer is C. This example is used to illustrate evidence organization rather than a performance improvement.
Fig. 6: Illustrative Social-IQ video-to-graph example. The video frames show participant grounding and temporally localized visual evidence relevant to the question. Participant identities are manually matched across frames for illustration. The reference answer is D, while the recorded model answer is unavailable. This example illustrates evidence organization rather than a verified model prediction.
Comparison (second minus first)
Social-IQ
EMER F1
EMER Exact
MDPE
Overall
Graph-SFT: compact − no graph
+7.00
+0.80
+6.02
+1.01
+6.25
THG-Omni: matched − no graph
+3.90
+1.63
+0.00
+6.03
+3.95
THG-Omni: matched − shuffled
+4.07
+4.97
+4.52
+7.03
+4.34
TABLE IV: IntentBench graph-input sensitivity, expressed as differences in percentage points between reported conditions. EMER uses sample-level F1 and exact set match; other task columns use accuracy.
Input
Exact
Set precision
Set recall
Macro F1
Micro F1
Question and options
14.67
37.90
56.67
42.18
44.23
Question, options, and graph
24.00
54.40
69.33
57.18
56.16
TABLE V: MER24 graph-information diagnostic on 150 samples. Both conditions exclude source video and audio; the only input difference is the predicted scene graph. All values are percentages. Set precision and recall are sample-level averages.
Fig. 7: NExT-QA temporal reasoning example from the validation set. Three video frames show the lady talking and the man’s subsequent reaction. SAM3 provides participant grounding, while the predicted graph organizes the temporal evidence used for answering the question. With the same Graph-SFT adapter, the no-graph prediction is C, whereas the matched-graph prediction is A, consistent with the reference answer.
Model / input
Causal
Temporal
Descriptive
Overall
CH
CW
TC
TN
TP
DC
DL
DO
Qwen2.5-Omni-7B [ 51 ]
76.21
79.18
77.51
72.34
75.27
61.18
93.79
85.50
77.98
Task-LoRA (no graph)
73.74
76.99
75.28
70.19
73.12
60.87
94.00
85.17
76.09
Graph-SFT (no graph)
76.39
79.18
77.25
71.77
74.19
61.18
93.58
85.67
77.86
Graph-SFT (matched graph)
81.66
83.21
82.59
76.65
80.72
70.39
95.11
88.86
82.40
THG-Omni (no graph)
83.79
85.56
84.80
81.20
83.91
75.99
96.13
90.96
85.10
TABLE VI: NExT-QA test accuracy (%) across reasoning types on 8,564 questions.
Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.
Yifan Shen, Pei Tian, Xinzhuo Li +8
University of Illinois Urbana-Champaign · 2Columbia University · 3Google
Recent video reasoning models increasingly produce spatio-temporal evidence chains that localize objects at specific timestamps. While these traces improve interpretability by grounding where and when evidence appears, they often leave the motion connecting observations, the how, implicit. This makes dynamic and trajectory-dependent claims difficult to supervise, verify, or penalize when unsupported by the video. We formalize this missing component as Spatial-Temporal-Trajectory (STT) reasoning and introduce Motion-o, a motion-centric extension to vision-language models (VLMs) that makes trajectories explicit and verifiable. Motion-o augments evidence chains with Motion Chain of Thought (MCoT), a structured pathway that represents object motion through a discrete <motion/> tag summarizing direction, speed, and scale change. To supervise MCoT, we densify sparse spatio-temporal annotations into object tracks and derive motion descriptors from centroid displacement and box-area change. We then train with complementary rewards for trajectory consistency and visual grounding, including a perturbation-based signal that penalizes motion descriptions that remain unchanged when temporal evidence is removed. Across multiple video understanding benchmarks, Motion-o consistently improves trajectory-faithful reasoning without architectural modifications. These results suggest that an explicit motion interface can complement existing VLM pipelines by converting implicit dynamics into verifiable evidence. Code is available at~\faGithub\ ostadabbas/Motion-o.
Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call -- a sparse global preview and a denser local clip -- we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by video editing and composition. Experiments across omnimodal and video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.
Yu Chen, Caorui Li, Ziyu Xiong +8
University of Chinese Academy of Sciences · Institute of Automation, CAS · Southeast University +3