PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition

Mingcong Ding; Babette Bühler; Tim Fütterer; Ulrich Trautwein; Enkelejda Kasneci

cs.CVOct 18, 2026

PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition

Authors: Mingcong Ding, Babette Bühler, Tim Fütterer, Ulrich Trautwein, Enkelejda Kasneci

Organizations: Technical University of Munich Munich, Germany · Munich Center for Machine Learning (MCML) Munich, Germany · University of Tübingen Tübingen, Germany

Abstract

Recognizing fine-grained bodily behaviors, such as self-grooming, postural shifts, and backchanneling, is a useful intermediate step for computational social signal analysis. But the behaviors that matter vary across social contexts, and retraining a model for each new taxonomy is expensive. Vision-language models (VLMs) can adapt to new categories through text prompts, yet they lack sensitivity to body structure and often confuse subtle postural distinctions. We propose PRIMA (Pose Refined Inductive Motion Adapter), a lightweight module that injects a body-structure prior into frozen VLMs using human pose as a training-time supervisory signal, requiring no pose input at inference. PRIMA separates a model-specific projection head from a shared core that encodes transferable body-configuration knowledge; adapting to a new VLM requires retraining only the head. We train PRIMA on a micro-action dataset and evaluate entirely through zero-shot prompting. The adapter improves recognition on its training domain, transfers to additional VLM backbones at different scales with the core frozen, and generalizes to two unseen target datasets from social psychology and daily activity recognition. In a supporting downstream study, the improved behavior predictions also yield higher correlations with human ratings of rapport and dominance in a collaboration context. Within the scope of the tested models and indoor datasets, the results indicate that pose-guided training can produce body-structure priors that generalize across unseen behavior vocabularies and can be partially reused across VLM backbones while preserving RGB-only inference.

Figures & tables

Figure 1. From social signals to transferable body structure priors. (a) Social attributes lack stable definitions. (b) Bodily behaviors are concrete, but (c) their label sets change with the setting. (d) PRIMA injects a pose-guided prior that transfers from RGB alone. Four-panel illustration. Panel a: two people talking with question-marked labels for rapport, disengagement, and interest. Panel b: the same scene relabelled with concrete bodily behaviors such as head nod, arm cross, and forward lean. Panel c: three separate settings (therapy, classroom, interview) each with its own behavior labels, showing that a supervised model must be retrained for every setting. Panel d: a vision-language model combined with PRIMA recognizing behaviors in all three settings from RGB alone.

Figure 2. PRIMA pipeline. Top : skeleton overlaid frames produce teacher features $\mathbf{E}_{t}$ (precomputed, cached). Middle : RGB frames pass through the trainable adapter to produce $\mathbf{E}_{s}^{\prime}$ ; $\mathcal{L}_{\text{align}}$ closes the gap. Bottom : inference uses only the student path with the adapter frozen. Three-row pipeline diagram showing teacher path, student path, and inference path.

Symbol	Description
$f_{v}$ , $f_{p}$	Frozen vision encoder and vision projector
$\mathcal{A}$	PRIMA adapter ( $=\mathcal{H}+\mathcal{C}$ )
$\mathcal{C}$	Core (model agnostic, operates in $\mathbb{R}^{d_{0}}$ )
$\mathcal{H}$	Projection Head (model specific)
$W_{\text{in}}$ , $W_{\text{out}}$	Linear projections in $\mathcal{H}$
$g$	Learnable gate scalar (init 0.01)

Table 1. Notation summary.

Figure 3. Adapter internal structure. Blue: Head

\mathcal{H}

(

d_{v}\leftrightarrow 768

, model specific). Purple: Core

\mathcal{C}

(two blocks, model agnostic). Gate

g

scales

\boldsymbol{\delta}

before addition to

\mathbf{V}_{s}

. Vertical flowchart of the adapter with Head and Core components.

Figure 4. Cross backbone transfer. Stage 1 trains the full adapter on Qwen3 VL. Stage 2 freezes the Core and trains only a new Head for InternVL3.5. Two-stage transfer diagram showing Core reuse across VLMs.

Method	micro MAP	macro MAP
Swin + PDAN (supervised) ( Balazia et al., 2022 )	0.745	0.374
Qwen3 VL baseline (zero-shot)	0.403	0.151
PRIMA (zero-shot)	0.448	0.185

Table 2. BBSI evaluation under the original frame-level MAP protocol ( Balazia et al., 2022 ) . Supervised results are from the original benchmark using Swin features with PDAN detection trained on BBSI labels. Our methods use zero-shot prompting with no BBSI training data.

		MMA 52 (source)		BBSI (transfer)		NTU (transfer)
Backbone	Method	$\text{F1}_{\text{pool}}$	$\text{F1}_{\text{cls}}$	$\text{F1}_{\text{pool}}$	$\text{F1}_{\text{cls}}$	$\text{F1}_{\text{pool}}$	$\text{F1}_{\text{cls}}$
Qwen3 VL 8B	RGB baseline	0.253	0.162	0.443	0.219	0.391	0.243
Qwen3 VL 8B	PRIMA	0.324	0.186	0.490	0.232	0.438	0.278
InternVL3.5 8B	RGB baseline	0.174	0.109	0.382	0.156	0.312	0.186
InternVL3.5 8B	PRIMA (frozen Core)	0.213	0.158	0.427	0.208	0.358	0.227
InternVL3.5 4B	RGB baseline	0.148	0.091	0.349	0.137	0.282	0.163

Table 3. Zero-shot recognition results. All methods use the same VLM and prompting protocol with no target-dataset labels. PRIMA uses pose only during training; at inference, it receives only RGB. Best result per backbone in bold ;

\pm

values across three seeds

\leq 0.005

on all settings. InternVL3.5 4B tests cross-scale transfer with the same frozen Core.

Loss components
Variant	$\text{F1}_{\text{pool}}$	$\text{F1}_{\text{cls}}$
PRIMA (full)	0.324	0.186
w/o cosine loss (MSE only)	0.305	0.169
w/o MSE loss (cosine only)	0.291	0.158
w/o regularization ( $\lambda=0$ )	0.313	0.175
Teacher signal

Table 4. Ablation on MMA 52 with Qwen3 VL. Each row modifies one component.

Dataset	Method	$\text{F1}_{\text{pool}}$	$\text{F1}_{\text{cls}}$
MMA 52	RGB baseline	0.253	0.162
	LoRA (supervised)	0.362	0.214
	Supervised adapter	0.341	0.198
	PRIMA (pose-guided)	0.324	0.186
BBSI	RGB baseline	0.443	0.219
	LoRA (supervised)	0.451	0.215

Table 5. Comparison with alternative adaptation methods on Qwen3 VL. All methods use the same zero-shot prompting protocol at evaluation. Supervised methods are trained on MMA 52 classification labels. PRIMA uses no classification labels.

Figure 5. Downstream social attribute prediction on BBSI (Qwen3 VL, Ridge, leave-one-recording-out CV). (a) PRIMA improves correlation with human ratings on all five attributes;

*

p<0.05

, paired permutation test. (b) Per-class F1 gain correlates with each behavior’s weight in the rapport regression (

r=0.71

p<0.01

). Hf: Hand-face, Hm: Hand-mouth, Lc: Legs crossed, Fa: Fold arms, Se: Settle, Sh: Shrug, Ge: Gesture, Gr: Groom, Sc: Scratch, Lm: Leg movement, St: Stretching, Ac: Adjusting clothing, Fu: Fumble, Sm: Smearing hands. Left: grouped bar chart comparing Spearman correlations for five social attributes between RGB baseline and PRIMA. Right: scatter plot of per-class F1 gain against absolute regression coefficient in the rapport model, showing positive correlation with confidence band.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

May 11, 2026

PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models

Large-scale pretraining has made Vision-Language-Action (VLA) models promising foundations for generalist robot manipulation, yet adapting them to downstream tasks remains necessary. However, the common practice of full fine-tuning treats pretraining as initialization and can shift broad priors toward narrow training-distribution patterns. We propose PriorVLA, a novel framework that preserves pretrained priors and learns to leverage them for effective adaptation. PriorVLA keeps a frozen Prior Expert as a read-only prior source and trains an Adaptation Expert for downstream specialization. Expert Queries capture scene priors from the pretrained VLM and motor priors from the Prior Expert, integrating both into the Adaptation Expert to guide adaptation. Together, PriorVLA updates only 25% of the parameters updated by full fine-tuning. Across RoboTwin 2.0, LIBERO, and real-world tasks, PriorVLA achieves stronger overall performance than full fine-tuning and state-of-the-art VLA baselines, with the largest gains under out-of-distribution (OOD) and few-shot settings. PriorVLA improves over pi0.5 by 11 points on RoboTwin 2.0-Hard and achieves 99.1% average success on LIBERO. Across eight real-world tasks and two embodiments, PriorVLA reaches 81% in-distribution (ID) and 57% OOD success with standard data. With only 10 demonstrations per task, PriorVLA reaches 48% ID and 32% OOD success, surpassing pi0.5 by 24 and 22 points, respectively.

Xinyu Guo, Bin Xie, Wei Chai +4

Institute of Automation, Chinese Academy of Sciences · Dexmal · Nanjing University of Aeronautics and Astronautics +1

VLA Model Adaptation Robot Policy Adaptation Robot Skill Learning OOD Generalization Vision-Language-Action Models

Ask PDF

May 27, 2026

PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation

Vision-Language-Action (VLA) models offer a promising paradigm for generalist robotic policies, yet their adaptation is hindered by data inefficiency and poor generalization. We argue that these bottlenecks stem from the prevailing Direct Instruction-to-Control Mapping, which forces models to memorize monolithic trajectories rather than reusable motion patterns, i.e., primitives. We propose PrimitiveVLA, a framework that shifts this paradigm toward a Primitive-Centric Disassemble & Assemble paradigm. Supported by a shared Multimodal Canonical Representation (MCR), PrimitiveVLA unifies two phases: (1) Fine-tuning-phase Disassembly, which uses an automated pipeline to disassemble demonstrations into reusable primitives; and (2) Inference-phase Assembly, which employs a VLM-based planner and an LLM-generated switch module for robust closed-loop execution. By disassembling tasks into reusable primitives, PrimitiveVLA enables VLA models to learn invariant motion patterns instead of task-specific trajectories. Extensive experiments show that our framework improves data efficiency and achieves superior zero-shot generalization across unseen and long-horizon tasks.

Yutai Li, Shaohui Peng, Jiaming Guo +8

State Key Lab of Processors, Institute of Computing Technology, CAS · Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, CAS · University of Chinese Academy of Sciences +3

Long-Horizon Robotic Manipulation Robot Skill Learning Generalization in Robotic Manipulation Robotic Manipulation Data-Efficient Robot Learning Vision-Language-Action Models+1

Ask PDF

Mar 12, 2026

BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning

Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.

Jingyang Ke, Weihan Li, Amartya Pradhan +2

Georgia Institute of Technology Atlanta, GA, 30332 · Georgia Institute of Technology, Emory University Atlanta, GA, 30322

Animal Behavior Analysis VLM Reasoning

Ask PDF

Abstract

VLM Adaptation

Explore similar work

PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models

PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation

BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning