Do the Details Help the Whole? One-Way Transfer from Person Actions to Group Intent in VLMs on JRDB-Social
Organizations: Independent Researcher · Monash University · University of Chinese Academy of Sciences · Pusan National University ·
Abstract
Social scene datasets label people at two levels: what each person is doing and what the group is there for. Training on both levels at once is often assumed to help each of them. We test this assumption for vision-language models (VLMs) on JRDB-Social, treating person actions as the details and group intent as the whole. We fine-tune Qwen3.5 4B under eight training arms with three seeds, repeat the core arms on Qwen3.5 9B and Qwen3-VL 4B, and judge every arm with a verdict rule fixed before testing. Writing each member's actions before the intent raises intent macro-F1 from 30.1 to 31.0, and writing the intent first reaches 31.2, while action F1 stays within 0.2 points of an action-only model. Under the selected recipe, no arm trained with intent improves actions. The gain requires correct, image-grounded actions: empty action lists or actions borrowed from another group with the same intent lower intent F1. Forced-prefix tests, occlusion, member masking, linear probes and component grafting show that the intent follows the written actions, relies more on people and partly resides in the vision tower. Supplying actions as input instead of as targets hurts both levels.
Figures & tables
| Person actions | Group intent | ||||||||
| Arm | F1 | F1 | Multi | Single | 5-class | Visible | Verdict | ||
| Zero shot | 34.8 | 26.5 | 23.6 | 25.9 | 37.5 | 29.1 | zero shot | ||
| S-F | 59.8 | – | – | – | – | – | reference | ||
| S-C | – | 30.1 | 31.6 | 25.0 | 48.7 | 31.8 | reference | ||
| MIX | 57.6 | 2.2 | 30.6 | +0.5 | 32.0 | 25.5 | 48.9 | 33.3 | Trades |
| J-FC | 59.7 | 0.1 | 31.0 | +0.9 | 31.6 | 25.9 | 48.3 | 33.4 | Trades |
| Setting | Arm | Verdict | ||
| Qwen3-VL 4B | MIX | 2.0 | +0.5 | Trades |
| J-FC | 0.8 | 1.2 | Harms | |
| J-CF | 1.7 | +0.0 | Trades | |
| COND | 4.2 | 1.1 | Harms | |
| Qwen3.5 9B † | MIX | 3.5 | +1.0 | Trades |
| J-FC | 1.5 | +1.5 | Trades |
| Measure | S-C | J-FC |
|---|---|---|
| Person share of the occlusion loss | 0.40 | 0.46 |
| Intent change, one member hidden | 0.163 | 0.194 |
| Marked member hit rate | 0.545 | 0.555 |
| chance level | 0.471 | 0.477 |
| Action probe gain (S-F: 2.7) | 0.5 | 2.6 |
| Intent probe gain (S-F: 1.1) | 2.0 | 1.8 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Test groups | Oracle F1 | Cohen’s on blind items | |||||
|---|---|---|---|---|---|---|---|
| Intent | all | multi | actions | actions + location | human vs. gold | human vs. Opus | Opus vs. gold |
| commuting | 624 | 90 | 83.6 | 79.4 | 0.61 | 0.71 | 0.45 |
| eating | 211 | 76 | 47.0 | 53.6 | 0.66 | 0.69 | 0.63 |
| socializing | 147 | 95 | 47.9 | 47.9 | 0.46 | 0.50 | 0.36 |
| navigating | 100 | 16 | 16.6 | 27.2 | 0.00 | 0.00 | |
| studying/working | 98 | 36 | 38.4 | 43.6 | 0.69 | 0.61 | 0.53 |
| Action | Intent | |||||
| Arm | 42 | 1 | 2 | 42 | 1 | 2 |
| S-F | 60.4 | 59.1 | 59.9 | – | – | – |
| S-C | – | – | – | 29.1 | 30.2 | 31.0 |
| MIX | 58.3 | 56.8 | 57.6 | 30.9 | 29.4 | 31.4 |
| J-FC | 59.5 | 58.5 | 61.1 | 31.0 | 30.5 | 31.5 |
| J-CF | 59.4 | 58.6 | 60.8 | 31.8 | 30.5 | 31.5 |
| Action | Intent | Multi-member groups | Single people | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Arm | multi | single | multi | single | verdict | verdict | ||||
| S-F | 56.8 | 52.1 | – | – | reference | reference | ||||
| S-C | – | – | 31.6 | 25.0 | reference | reference | ||||
| MIX | 54.2 | 50.6 | 32.0 | 25.5 | Trades | Trades | ||||
| J-FC | 56.9 | 50.6 | 31.6 | 25.9 | Trades | Trades | ||||
| J-CF | 56.0 | 52.0 | 31.4 | 26.1 | Harms | Trades | ||||
| Backbone | Arm | ||||
| Qwen3.5 9B | zero shot | 36.1 | 17.8 | ||
| (seed 42) | S-F | 57.6 | – | ||
| S-C | – | 26.5 | |||
| MIX | 54.0 | 27.4 | |||
| J-FC | 56.0 | 28.0 | |||
| COND | 52.2 | 23.8 |
| Arm | com. | eat. | soc. | nav. | stu. | dis. | wai. |
|---|---|---|---|---|---|---|---|
| S-C | 80.4 | 64.9 | 29.8 | 6.1 | 48.3 | 0.0 | 11.3 |
| J-FC | 80.6 | 66.4 | 23.6 | 9.9 | 46.3 | 6.8 | 14.3 |
| J-CF | 80.1 | 65.3 | 25.8 | 8.8 | 48.4 | 7.0 | 14.6 |
| MIX | 81.1 | 66.5 | 24.6 | 4.9 | 48.9 | 5.0 | 13.5 |
| J-sFC | 77.3 | 63.0 | 24.9 | 7.6 | 48.9 | 9.5 | 12.2 |
| COND | 79.6 | 64.6 | 15.0 | 3.6 | 47.0 | 2.7 | 9.6 |
| Opus recovers it from | Pairs | S-C | J-FC | J-CF | MIX |
|---|---|---|---|---|---|
| people only | 101 | 55.8 | |||
| people and place | 338 | 95.7 | |||
| place only | 247 | 89.3 | |||
| neither | 89 | 55.1 | |||
| not even the full view | 543 | 23.4 |
| Arm | Actions | All | Multi | Single | 5-cl. | Vis. |
|---|---|---|---|---|---|---|
| S-C | none | 30.1 | 31.6 | 25.0 | 48.7 | 31.8 |
| J-nFC | empty lists | 29.8 | 30.9 | 24.4 | 46.9 | 31.3 |
| J-aFC | donor, same intent | 28.1 | 30.0 | 23.6 | 45.2 | 30.4 |
| J-sFC | donor, other intent | 30.4 | 31.5 | 25.0 | 47.7 | 33.4 |
| J-FC | correct | 31.0 | 31.6 | 25.9 | 48.3 | 33.4 |
| same day | other day | unseen | |
| Arm | (651) | (58) | (427) |
| S-C | 28.4 | 23.0 | 31.5 |
| MIX | 27.8 | 28.6 | 31.7 |
| J-FC | 28.2 | 33.2 | 30.5 |
| J-CF | 28.9 | 30.4 | 30.6 |
| COND | 25.7 | 23.9 | 28.3 |
| Arm | multi | single | 5-class | ||
|---|---|---|---|---|---|
| AG-F | 60.0 | – | – | – | – |
| AG-C | – | 30.9 | 32.7 | 24.6 | 48.3 |
| AG-FC | 59.7 | 30.4 | 31.0 | 25.6 | 49.4 |
| AG-CF | 58.4 | 28.8 | 30.3 | 23.4 | 44.8 |
| Direction | Replaced | Correct | Wrong | Empty |
|---|---|---|---|---|
| actions intent | final turn | |||
| all turns | ||||
| intent actions | final turn | |||
| all turns |
| train+dev prompts | dev prompts | |||
|---|---|---|---|---|
| Arm | ||||
| RC | ||||
| RF | ||||
| RFC | ||||
| RFC-gated | ||||
| Run | varying | ||
|---|---|---|---|
| RC, seen | 0.867 0.889 | 0.43 0.30 | |
| RF, seen | 0.871 0.916 | 0.52 0.39 | |
| RFC, seen | 0.871 0.884 | 0.874 0.905 | 0.69 0.59 |
| RC, dev | 0.684 0.744 | 0.53 0.36 | |
| RF, dev | 0.717 0.765 | 0.70 0.48 | |
| RFC, dev | 0.677 0.738 | 0.715 0.754 | 0.82 0.64 |
| Arm | Intent change | Gold-intent drop | Others | Pairs | Marked | Chance | Strong | Weak |
| S-F | – | – | 0.216 | – | – | – | – | – |
| S-C | 0.163 | 0.056 | – | 162 | 0.545 | 0.471 | 0.190 | 0.159 |
| J-FC | 0.194 | 0.067 | 0.188 | 179 | 0.555 | 0.477 | 0.233 | 0.197 |
| J-CF | 0.177 | 0.050 | 0.229 | 151 | 0.535 | 0.478 | 0.202 | 0.159 |
| MIX | 0.174 | 0.054 | – | 170 | 0.567 | 0.480 | 0.186 | 0.171 |
| COND | 0.246 | 0.103 | – | 212 | 0.470 | 0.469 | 0.232 | 0.258 |
| Arm | 42 | 1 | 2 | Mean |
|---|---|---|---|---|
| zero shot | 0.42 | 0.42 | 0.42 | |
| S-C | 0.29 | 0.47 | 0.43 | 0.40 |
| MIX | 0.60 | 0.44 | 0.47 | 0.50 |
| J-FC | 0.46 | 0.45 | 0.48 | 0.46 |
| J-CF | 0.49 | 0.44 | 0.60 | 0.51 |
| COND | 0.71 | 0.43 | 0.64 | 0.59 |
| Direction | Prefix | 42 | 1 | 2 |
|---|---|---|---|---|
| actions intent | correct | |||
| ( J-FC ) | wrong | |||
| empty | ||||
| intent actions | correct | |||
| ( J-CF ) | wrong | |||
| empty |
| Intent | Action | ||||
| Arm | Prompt | best | late | best | late |
| S-F | J-FC | 1.1 | 2.6 | 2.7 | 2.0 |
| S-C | J-FC | 2.0 | 2.7 | 0.5 | 0.1 |
| MIX | J-FC | 2.0 | 3.1 | 2.6 | 2.5 |
| J-FC | J-FC | 1.8 | 3.3 | 2.6 | 2.7 |
| J-CF | J-FC | 2.5 | 3.4 | 3.3 | 2.9 |
Intent Classification
Intent classification assigns user utterances or behaviors to goals, supporting dialogue systems, search, safety decisions, and proactive assistance.
See all 65 papers