THG-Omni: Traceable Human-Centric Graph Supervision for Video Reasoning
Authors: Jisheng Dang, Kai Wei, Ronghao Lin, Bimei Wang, Hong Peng, Wei-Shi Zheng, Bin Hu, Qi Tian, Tat-Seng Chua
Organizations: School of Information Science & Engineering, Lanzhou University, Lanzhou 730000, China. · the School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China · the School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China · Cloud and AI BU, Huawei, Shenzhen, Guangdong 518129, China · the School of Computing, National University of Singapore, Singapore 119077
Omni-modal models can answer questions about video, but their predictions rarely show which participant, event, or interval supports an answer. Moreover, answer-level supervision alone does not teach a model to interpret structured video evidence. To systematically mitigate this issue, we present THG-Omni, a human-centric graph-supervision framework that links Segment Anything Model~3 participant tracks with temporally grounded Qwen3-VL annotations. Candidate semantic claims are checked against the source video and available audio before they are used to construct training questions, while track geometry provides explicit spatial and temporal supervision. The resulting questions teach an omni-modal reasoning model to interpret participant attributes, actions, affect, interactions, and their timing from a structured graph. At inference, THG-Omni uses a predicted graph constructed from the evaluation video. Experiments on IntentBench show consistent improvements over the unadapted backbone across social reasoning, emotion understanding, and deception perception. On NExT-QA, THG-Omni achieves 89.54% accuracy with the matched graph, compared with 85.10% without graph input. Further validation experiments show that compact graph input achieves 78.22% accuracy, compared with 78.02% using the full graph and 77.58% without graph input. A complementary diagnostic demonstrates that the predicted graph preserves evidence useful for answering questions even without the source video and audio. Together, these findings show that THG-Omni improves human-centric video reasoning while grounding its evidence in identifiable participants and events. Our codes are available at https://anonymous.4open.science/r/review-n94hhfcwp5666/
Figures & tables
Fig. 1: Motivation for traceable graph supervision. This conceptual example contrasts answer-level supervision with identity- and time-grounded graph supervision. The depicted participants and event illustrate the learning objective rather than a benchmark prediction.
Fig. 2: Overview of the proposed traceable human-centric graph-supervision framework. (a) Temporally ordered video evidence. (b) SAM3 grounds and tracks participants, while Qwen3-VL annotates visible attributes, emotions, actions, interactions, and their temporal intervals. (c) The grounded evidence forms a video graph with identity-bound nodes, temporal states, and participant relations. (d) HumanOmniV2 verifies each candidate claim using the source video and available audio; only supported claims are retained. (e) The verified graph supports training of THG-Omni, which is evaluated with video-matched, shuffled, and omitted graph input.
1:
Initialize Dgraph←∅ .
2:
for each training video V and grounding question Q do
3:
Obtain tracks T←gSAM3(V,p(Q)) .
4:
Construct GT=(VT,ET) from track statistics and spatial relations.
5:
Generate semantic claims:
S←gann(V,contact(V),H(T),LC) .
6:
Review identity binding, temporal extent, label validity, and visual support.
Table 3
Fig. 3: IntentBench results and MER24 graph diagnostic. (a) Qwen2.5-Omni-7B and video-matched THG-Omni are compared on 2,688 questions. (b) On 150 MER24 samples without source video or audio, graph input improves exact set match over text-only input.
Social-IQ
EMER
MDPE
Overall
Model / input
Accuracy
Sample F1
Exact
Accuracy
Score
LLaVA-NeXT-Video [ 50 ]
31.58
29.68
12.03
39.20
32.05
HumanOmniV2 [ 7 ]
51.57
80.03
54.89
53.77
53.14
Qwen2.5-Omni-7B [ 51 ]
53.99
68.09
41.35
61.81
55.27
Graph-SFT (w/o graph)
49.66
70.52
40.60
60.30
51.48
Graph-SFT (compact graph)
56.66
71.32
46.62
61.31
57.73
TABLE I: IntentBench results on 2,688 questions. Overall is the sample-weighted aggregate of Social-IQ accuracy, EMER sample F1, and MDPE accuracy. LLaVA-NeXT-Video uses video without audio. Graph-SFT uses a distinct adapter; the THG-Omni rows report different graph-input conditions for the same trained model.
Method
Why (698)
How (632)
What (480)
When (14)
Who/Which (25)
Other (507)
Overall
Qwen2.5-Omni-7B [ 51 ]
53.44
54.11
53.54
57.14
64.00
54.44
53.99
Graph-SFT (w/o graph)
47.99
49.05
50.00
50.00
56.00
52.07
49.66
Graph-SFT (compact graph)
55.01
56.01
57.08
57.14
64.00
58.97
56.66
THG-Omni (w/o graph)
66.19
65.66
66.88
64.29
72.00
71.01
67.28
THG-Omni (shuffled graph)
66.48
65.19
65.83
64.29
72.00
71.40
67.11
THG-Omni (matched graph)
69.63
69.78
70.83
71.43
80.00
74.95
71.18
TABLE II: Social-IQ accuracy (%) across six question types on 2,356 questions.
Fig. 4: Illustrative Social-IQ video-to-graph example. The video frames show two participants and their interaction during a card game. SAM3 provides person tracks and geometric evidence, while semantic annotations describe the observed interaction. The example illustrates participant grounding and temporally localized visual evidence associated with the Social-IQ question. The reference answer is B.
Method
Precision
Recall
Sample F1
Exact
Qwen2.5-Omni-7B [ 51 ]
74.00
68.53
68.09
41.35
Graph-SFT (w/o graph)
76.20
70.10
70.52
40.60
Graph-SFT (compact graph)
78.10
70.80
71.32
46.62
THG-Omni (w/o graph)
89.54
83.97
84.63
66.17
THG-Omni (shuffled graph)
87.97
79.52
81.29
61.65
THG-Omni (matched graph)
93.36
83.91
86.26
66.17
TABLE III: EMER multi-label performance on 133 examples (%).
Fig. 5: Illustrative Social-IQ video-to-graph example. The video frames show the visible participants and the woman’s later reaction. The example illustrates participant grounding and temporally localized visual evidence associated with the Social-IQ question. The reference answer is C. This example is used to illustrate evidence organization rather than a performance improvement.
Fig. 6: Illustrative Social-IQ video-to-graph example. The video frames show participant grounding and temporally localized visual evidence relevant to the question. Participant identities are manually matched across frames for illustration. The reference answer is D, while the recorded model answer is unavailable. This example illustrates evidence organization rather than a verified model prediction.
Comparison (second minus first)
Social-IQ
EMER F1
EMER Exact
MDPE
Overall
Graph-SFT: compact − no graph
+7.00
+0.80
+6.02
+1.01
+6.25
THG-Omni: matched − no graph
+3.90
+1.63
+0.00
+6.03
+3.95
THG-Omni: matched − shuffled
+4.07
+4.97
+4.52
+7.03
+4.34
TABLE IV: IntentBench graph-input sensitivity, expressed as differences in percentage points between reported conditions. EMER uses sample-level F1 and exact set match; other task columns use accuracy.
Input
Exact
Set precision
Set recall
Macro F1
Micro F1
Question and options
14.67
37.90
56.67
42.18
44.23
Question, options, and graph
24.00
54.40
69.33
57.18
56.16
TABLE V: MER24 graph-information diagnostic on 150 samples. Both conditions exclude source video and audio; the only input difference is the predicted scene graph. All values are percentages. Set precision and recall are sample-level averages.
Fig. 7: NExT-QA temporal reasoning example from the validation set. Three video frames show the lady talking and the man’s subsequent reaction. SAM3 provides participant grounding, while the predicted graph organizes the temporal evidence used for answering the question. With the same Graph-SFT adapter, the no-graph prediction is C, whereas the matched-graph prediction is A, consistent with the reference answer.
Model / input
Causal
Temporal
Descriptive
Overall
CH
CW
TC
TN
TP
DC
DL
DO
Qwen2.5-Omni-7B [ 51 ]
76.21
79.18
77.51
72.34
75.27
61.18
93.79
85.50
77.98
Task-LoRA (no graph)
73.74
76.99
75.28
70.19
73.12
60.87
94.00
85.17
76.09
Graph-SFT (no graph)
76.39
79.18
77.25
71.77
74.19
61.18
93.58
85.67
77.86
Graph-SFT (matched graph)
81.66
83.21
82.59
76.65
80.72
70.39
95.11
88.86
82.40
THG-Omni (no graph)
83.79
85.56
84.80
81.20
83.91
75.99
96.13
90.96
85.10
TABLE VI: NExT-QA test accuracy (%) across reasoning types on 8,564 questions.