Organizations: Independent Researcher · Monash University · University of Chinese Academy of Sciences · Pusan National University · Shenzhen University of Advanced Technology
Social scene datasets label people at two levels: what each person is doing and what the group is there for. Training on both levels at once is often assumed to help each of them. We test this assumption for vision-language models (VLMs) on JRDB-Social, treating person actions as the details and group intent as the whole. We fine-tune Qwen3.5 4B under eight training arms with three seeds, repeat the core arms on Qwen3.5 9B and Qwen3-VL 4B, and judge every arm with a verdict rule fixed before testing. Writing each member's actions before the intent raises intent macro-F1 from 30.1 to 31.0, and writing the intent first reaches 31.2, while action F1 stays within 0.2 points of an action-only model. Under the selected recipe, no arm trained with intent improves actions. The gain requires correct, image-grounded actions: empty action lists or actions borrowed from another group with the same intent lower intent F1. Forced-prefix tests, occlusion, member masking, linear probes and component grafting show that the intent follows the written actions, relies more on people and partly resides in the vision tower. Supplying actions as input instead of as targets hurts both levels.
Figures & tables
Figure 1: Three people at a bus stop seen by a mobile robot. A model trained only on group intent guesses commuting ; a model trained to first write each member’s actions answers waiting .
Figure 2: Overview of the study. (a) The training arms differ only in which levels appear as targets and in which order. (b) Controls on the joint target, the prompt, the learning rate, the intent labels and the training steps. (c) Other backbones, a tool-using agent and GRPO rewards. (d) Analyses inside the trained models.
Person actions
Group intent
Arm
F1
ΔF
F1
ΔC
Multi
Single
5-class
Visible
Verdict
Zero shot
34.8
26.5
23.6
25.9
37.5
29.1
zero shot
S-F
59.8
–
–
–
–
–
reference
S-C
–
30.1
31.6
25.0
48.7
31.8
reference
MIX
57.6
− 2.2
30.6
+0.5
32.0
25.5
48.9
33.3
Trades
J-FC
59.7
− 0.1
31.0
+0.9
31.6
25.9
48.3
33.4
Trades
Table 1: Main results on the JRDB-Social test set with Qwen3.5-4B (macro-F1 ×100 , mean of three seeds). ΔF and ΔC compare each arm with S-F and S-C. Multi and Single split the intent score by group size, 5-class merges the intent labels into five families, and Visible restricts it to the 336 groups whose intent can be judged from the frames (§ 4 ).
Figure 3: Intent macro-F1 when intent labels are kept for 25%, 50% or all training groups (seed 42; two draws per budget, averaged). Action labels are always complete.
Figure 4: Intent macro-F1 (three seeds) when the action part of the joint target is empty (J-nFC), taken from another group with the same intent (J-aFC), taken from another group with a different intent (J-sFC), or correct (J-FC).
Figure 5: Per-class intent F1 of S-C and J-FC , averaged over three seeds. Leisure is 0 for both and is not shown.
Setting
Arm
ΔF
ΔC
Verdict
Qwen3-VL 4B
MIX
− 2.0
+0.5
Trades
J-FC
− 0.8
− 1.2
Harms
J-CF
− 1.7
+0.0
Trades
COND
− 4.2
− 1.1
Harms
Qwen3.5 9B †
MIX
− 3.5
+1.0
Trades
J-FC
− 1.5
+1.5
Trades
Table 2: The transfer outside the main setting ( ×100 ). Deltas use the single-task arms of the same setting as references; the agent rows compare with AG-F and AG-C. † One seed. ‡ Two seeds, deltas relative to the J-FC starting point.
Figure 6: Forced prefixes. Blue: the model’s action field is replaced by the correct actions, another group’s actions or empty lists, and the intent is decoded. Orange: the intent field is replaced and the actions are decoded. Direct generation uses J-FC and J-CF (three seeds); the agent uses AG-FC and AG-CF with all earlier notes rewritten (seed 42).
Measure
S-C
J-FC
Person share ρ of the occlusion loss
0.40
0.46
Intent change, one member hidden
0.163
0.194
Marked member hit rate
0.545
0.555
chance level
0.471
0.477
Action probe gain (S-F: 2.7)
0.5
2.6
Intent probe gain (S-F: 1.1)
2.0
1.8
Table 3: How the two models use people and members. ρ is a three-seed mean; the masking and probe rows use seed 42. The hit rate is the share of the human-marked member among the members whose removal deletes a correct intent. Probe gains are balanced accuracy ×100 over the zero-shot model at the best layer, averaged over eight targets.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Test groups
Oracle F1
Cohen’s κ on blind items
Intent
all
multi
actions
actions + location
human vs. gold
human vs. Opus
Opus vs. gold
commuting
624
90
83.6
79.4
0.61
0.71
0.45
eating
211
76
47.0
53.6
0.66
0.69
0.63
socializing
147
95
47.9
47.9
0.46
0.50
0.36
navigating
100
16
16.6
27.2
0.00
0.00
−0.04
studying/working
98
36
38.4
43.6
0.69
0.61
0.53
Appendix
Table 4: The eight headline intents. Oracle F1: a logistic regression on the gold actions and the group size, alone or with the location. κ : 89 blind items (59 multi-member, 30 single); the human never used the label navigating on them, so its κ of 0 is degenerate.
Action F
Intent C
Arm
42
1
2
42
1
2
S-F
60.4
59.1
59.9
–
–
–
S-C
–
–
–
29.1
30.2
31.0
MIX
58.3
56.8
57.6
30.9
29.4
31.4
J-FC
59.5
58.5
61.1
31.0
30.5
31.5
J-CF
59.4
58.6
60.8
31.8
30.5
31.5
Appendix
Table 5: Scores ( ×100 ) of each Qwen3.5 4B arm for seeds 42, 1 and 2. The content controls write donor or empty actions by design, so their action scores are not meaningful (4.4 to 4.7 for J-sFC , below the majority prior).
Action F
Intent C
Multi-member groups
Single people
Arm
multi
single
multi
single
ΔF
ΔC
verdict
ΔF
ΔC
verdict
S-F
56.8
52.1
–
–
reference
reference
S-C
–
–
31.6
25.0
reference
reference
MIX
54.2
50.6
32.0
25.5
−2.5
+0.4
Trades
−1.5
+0.5
Trades
J-FC
56.9
50.6
31.6
25.9
+0.2
−0.1
Trades
−1.5
+0.9
Trades
J-CF
56.0
52.0
31.4
26.1
−0.8
−0.2
Harms
−0.1
+1.1
Trades
Appendix
Table 6: Three-seed means of Qwen3.5 4B split by group size, with the verdict rule applied inside each stratum (250 multi-member groups, 886 single people).
Backbone
Arm
F
C
ΔF
ΔC
Qwen3.5 9B
zero shot
36.1
17.8
(seed 42)
S-F
57.6
–
S-C
–
26.5
MIX
54.0
27.4
−3.5
+1.0
J-FC
56.0
28.0
−1.5
+1.5
COND
52.2
23.8
−5.4
−2.7
Appendix
Table 7: Absolute scores on the other two backbones. Deltas are computed before rounding.
Arm
com.
eat.
soc.
nav.
stu.
dis.
wai.
S-C
80.4
64.9
29.8
6.1
48.3
0.0
11.3
J-FC
80.6
66.4
23.6
9.9
46.3
6.8
14.3
J-CF
80.1
65.3
25.8
8.8
48.4
7.0
14.6
MIX
81.1
66.5
24.6
4.9
48.9
5.0
13.5
J-sFC
77.3
63.0
24.9
7.6
48.9
9.5
12.2
COND
79.6
64.6
15.0
3.6
47.0
2.7
9.6
Appendix
Table 8: Per-class intent F1, three-seed means. Leisure is 0 for every arm. The last row is the action oracle of Table 4 .
Opus recovers it from
Pairs
S-C
J-FC
J-CF
MIX
people only
101
55.8
−3.3
−0.3
+1.0
people and place
338
95.7
−1.5
−0.9
+0.3
place only
247
89.3
+1.2
+0.4
+0.9
neither
89
55.1
−1.1
−4.5
−3.0
not even the full view
543
23.4
−0.1
+0.1
+0.1
Appendix
Table 9: Recall of gold (group, intent) pairs, split by the views from which Opus ranks that intent first, three-seed means. Arm columns are differences from S-C .
Arm
Actions
All
Multi
Single
5-cl.
Vis.
S-C
none
30.1
31.6
25.0
48.7
31.8
J-nFC
empty lists
29.8
30.9
24.4
46.9
31.3
J-aFC
donor, same intent
28.1
30.0
23.6
45.2
30.4
J-sFC
donor, other intent
30.4
31.5
25.0
47.7
33.4
J-FC
correct
31.0
31.6
25.9
48.3
33.4
Appendix
Table 10: Intent macro-F1 ( ×100 , three seeds) when the action part of the joint target is emptied or replaced by the actions of a donor group, mostly from another location. All arms share the J-FC instruction and output format except S-C.
same day
other day
unseen
Arm
(651)
(58)
(427)
S-C
28.4
23.0
31.5
MIX
27.8
28.6
31.7
J-FC
28.2
33.2
30.5
J-CF
28.9
30.4
30.6
COND
25.7
23.9
28.3
Appendix
Table 11: Intent F1 by test location (three seeds; number of groups in brackets): same day and location as a training sequence, a training location on another day, or an unseen location. The last two rows give action F1.
Arm
F
C
multi
single
5-class
AG-F
60.0
–
–
–
–
AG-C
–
30.9
32.7
24.6
48.3
AG-FC
59.7
30.4
31.0
25.6
49.4
AG-CF
58.4
28.8
30.3
23.4
44.8
Appendix
Table 12: Agent arms on Qwen3.5 4B, three seeds. The last three columns split the intent score.
Direction
Replaced
Correct
Wrong
Empty
actions → intent
final turn
+1.7
−3.1
−1.9
all turns
+1.9
−2.4
−2.3
intent → actions
final turn
+1.8
−3.4
−1.9
all turns
+1.4
−2.5
−0.5
Appendix
Table 13: Forced prefixes in the agent (seed 42), relative to the forced own answer. With only the final turn replaced, the first impression of AG-CF still states an intent, and an empty intent then costs the actions 1.9 points.
train+dev prompts
dev prompts
Arm
ΔF
ΔC
ΔF
ΔC
RC
−0.1
+0.8
+0.1
−1.8
RF
−1.1
0.0
−2.5
+0.1
RFC
−1.0
+0.1
−2.0
−3.3
RFC-gated
−1.0
−0.4
+0.3
−2.7
Appendix
Table 14: GRPO changes relative to the starting model. Left: start from J-FC trained on train and development data, RL on the same prompts (two seeds). Right: start from J-FC trained on the training split only (57.4 / 29.6), RL on development prompts (seed 42).
Run
RC
RF
varying
RC, seen
0.867 → 0.889
0.43 → 0.30
RF, seen
0.871 → 0.916
0.52 → 0.39
RFC, seen
0.871 → 0.884
0.874 → 0.905
0.69 → 0.59
RC, dev
0.684 → 0.744
0.53 → 0.36
RF, dev
0.717 → 0.765
0.70 → 0.48
RFC, dev
0.677 → 0.738
0.715 → 0.754
0.82 → 0.64
Appendix
Table 15: Training rewards, first 16 steps → last 16 steps (seed 42), and the share of prompt groups whose samples differ in reward. “Seen”: RL prompts from the supervised training data; “dev”: unseen development prompts; “agent”: GRPO from AG-FC .
Arm
Intent change
Gold-intent drop
Others
Pairs
Marked
Chance
Strong
Weak
S-F
–
–
0.216
–
–
–
–
–
S-C
0.163
0.056
–
162
0.545
0.471
0.190
0.159
J-FC
0.194
0.067
0.188
179
0.555
0.477
0.233
0.197
J-CF
0.177
0.050
0.229
151
0.535
0.478
0.202
0.159
MIX
0.174
0.054
–
170
0.567
0.480
0.186
0.171
COND
0.246
0.103
–
212
0.470
0.469
0.232
0.258
Appendix
Table 16: Hiding one member (seed 42, 1,926 member and sample pairs). Intent change: change of the predicted intent set. Gold-intent drop: loss of set F1 against the gold intent. Pairs: (group, intent) pairs with a marked member that enter the hit rate. Marked: share of the marked member among the members whose removal deletes a correct intent. Strong and weak: intent change when a member rated as strong or weak evidence is hidden.
Figure 7: Linear probe balanced accuracy per layer, averaged over eight intent targets (left) and eight action targets (right). Seed 42; every model reads the J-FC prompt.
Figure 8: Component grafting (seed 42). Bars: change in intent F1 when one component group of the donor replaces that of the base model; the vision tower is shown in blue. Dots: three random perturbations of the same tensors with the same norm.
Arm
42
1
2
Mean
zero shot
0.42
0.42
0.42
S-C
0.29
0.47
0.43
0.40
MIX
0.60
0.44
0.47
0.50
J-FC
0.46
0.45
0.48
0.46
J-CF
0.49
0.44
0.60
0.51
COND
0.71
0.43
0.64
0.59
Appendix
Table 17: Person share ρ of the occlusion loss per seed.
Direction
Prefix
42
1
2
actions → intent
correct
+2.9
+1.3
+2.7
( J-FC )
wrong
−2.4
−4.0
−3.2
empty
−4.1
−3.7
−22.6
intent → actions
correct
+1.0
+1.6
+1.3
( J-CF )
wrong
−3.4
−3.4
−3.0
empty
0.0
−0.6
−0.8
Appendix
Table 18: Forced prefixes in direct generation per seed, change of the decoded side’s macro-F1 ( ×100 ).
Intent
Action
Arm
Prompt
best
late
best
late
S-F
J-FC
1.1
2.6
2.7
2.0
S-C
J-FC
2.0
2.7
0.5
0.1
MIX
J-FC
2.0
3.1
2.6
2.5
J-FC
J-FC
1.8
3.3
2.6
2.7
J-CF
J-FC
2.5
3.4
3.3
2.9
Appendix
Table 19: Probe gain over the zero-shot model (balanced accuracy ×100 , seed 42) at the best layer and averaged over layers 16 to 32. The last two rows feed each single-task model its own prompt and compare with the zero-shot model under the same prompt.