Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognition

Changyi Li; Yu Xiao

cs.CVSep 4, 2026

Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognition

Authors: Changyi Li, Yu Xiao

Organizations: Department of Information and Communications Engineering Aalto University 02150 Espoo, Finland

Abstract

Assembly actions are compositional: they combine a manipulation with a part or tool. In deployment, systems routinely encounter novel combinations of familiar components, yet an atomic action classifier assigns every unseen combination exactly zero probability by construction. The prevailing solution is verb--noun decomposition, which predicts components separately and recombines them to reach unseen actions. While widely adopted, how decomposition generalizes under compositional shift remains poorly understood. We present a systematic analysis of verb--noun decomposition across three assembly datasets (MECCANO, HAViD, and IMPACT). Although decomposition escapes the atomic ceiling, its generalization extends only partially beyond it. Unseen-composition performance remains strongly tied to the co-occurrence structure of the training data, indicating that much of the observed gain arises from interpolation within densely supported regions of the compositional space rather than from unconstrained recombination. Across datasets, failures consistently concentrate on the larger-vocabulary component, and IMPACT's verb-heavy vocabulary reverses the bottleneck from nouns to verbs. We further show that shared-encoder training introduces component entanglement, encouraging reliance on co-occurrence patterns that transfer poorly to unseen compositions and trailing independent recombination by up to $6.0\times$ in harmonic mean. Taken together, these findings explain why decomposition achieves only partial compositional generalization in practice. By identifying primitive support, vocabulary asymmetry, and component entanglement as connected sources of error, we provide a portable diagnostic framework for studying compositional recognition beyond aggregate accuracy. Code: https://github.com/hisalaheiyo/assembly.

Figures & tables

Figure 1: Recognition formulations studied in this paper. Top: atomic prediction ; middle: shared decomposition ; bottom: independent decomposition . Atomic prediction treats each action as a single class, while the decomposition variants predict verbs and nouns separately using either a shared or independent backbone and combine them via $\log P(v)+\log P(n)$ . The coloured output boxes indicate the reachable output space: atomic prediction is limited to training actions, whereas decomposition spans the full verb $\times$ noun space and can represent unseen compositions.

Dataset	$\|\mathcal{V}\|$	$\|\mathcal{N}\|$	grid	obs. (sparsity)	larger vocab	viewpoints / hands
MECCANO	12	23	276	61 (22.1%)	noun ( $1.9\times$ )	egocentric, single-hand
HAViD	11	29	319	165 (51.7%)	object ( $2.6\times$ )	3 static, bimanual
IMPACT	22	18	396	136 (34.3%)	verb ( $1.2\times$ )	5 views (incl. ego), bimanual

Table 1: Structural diversity of the datasets used in this study. IMPACT uniquely reverses the common noun-heavy regime, enabling a direct test of whether the bottleneck follows vocabulary size rather than component type.

Dataset	Config	Seen	Unseen	HM
MECCANO	Atomic	48.9	0.0	0.0
	Shared	48.4	1.7	3.3
	Independent	44.3	12.8	19.8
	C2C-shared	27.9	13.2	17.9
	C2C-indep	46.6	16.5	24.3
HAViD	Atomic	54.7	0.0	0.0

Table 2: Performance on the compositional split. Seen and unseen action top-1 accuracy and harmonic mean (HM) for five recognition formulations across three datasets. Atomic prediction serves as the unreachable baseline, while shared and independent decomposition make unseen compositions reachable. This table provides the reference point for all subsequent analyses.

Figure 2: Unseen accuracy increasingly follows primitive support as the composition grid becomes denser. Each point is an unseen verb–noun pair. Correlation strength rises from MECCANO to IMPACT to HAViD, mirroring the density of the observed composition grid. High-support pairs achieve the strongest performance, whereas low-support pairs remain near zero.

Figure 3: The bottleneck follows vocabulary size rather than component identity. (a) Unseen errors partitioned by component. Failures consistently concentrate on the larger-vocabulary component. (b) The verb bottleneck in IMPACT persists across all five views, indicating that the asymmetry follows vocabulary structure rather than viewpoint.

Dataset	$\|\mathcal{V}\|$	$\|\mathcal{N}\|$	both	obj err ( $\pi_{v}$ )	verb err ( $\pi_{n}$ )	both wrong	bottleneck
MECCANO	12	23	12.8	33.5	26.2	27.5	noun/obj
HAViD	11	29	18.6	39.6	12.9	28.9	noun/obj
IMPACT	22	18	22.1	15.1	32.6	30.2	verb
IMPACT per view (verb bottleneck is view-invariant):
front			24.5	15.1	31.8	28.6	verb
left			25.1	17.1	30.8	27.1	verb

Table 3: The bottleneck follows vocabulary size rather than component identity. Across datasets, unseen failures consistently concentrate on the larger-vocabulary component. IMPACT reverses the common noun-heavy regime, and the bottleneck reverses with it across all five views.

	shortcut $\Delta$ (pp)			grid	seen-margin
Dataset	verb	object	$\rho$ (D2)	density	(v, o) pp
MECCANO	6.4	11.5	0.49	22.1%	(-0.3, -0.6)
HAViD	5.9	8.7	0.69	51.7%	(-1.2, -1.9)
IMPACT	12.5	14.5	0.56	34.3%	(+1.8, +0.7)
$\Delta=$ excess seen $\to$ unseen drop of joint over solo (shortcut). seen-margin $\approx 0$ rules out capacity.

Table 4: Evidence for component entanglement in shared training. The shortcut score

\Delta

measures the excess seen-to-unseen drop of a jointly trained component head relative to its independently trained counterpart. Large

\Delta

values together with near-zero seen margins indicate that shared training provides little benefit on seen compositions while substantially increasing the unseen generalization gap.

Figure 4: More factorization does not imply better compositional generalization. HAViD performance under

k=1\ldots 4

factorization levels. The standard two-component decomposition (

k=2

) achieves the best unseen performance. Further factorization recovers seen accuracy but degrades generalization. Inset: the corresponding

k{=}1\!\to\!2

transition on MECCANO and IMPACT.

Figure 5: Interventions change performance levels but not the bottleneck. Unseen HM for shared and independent formulations, with and without C2C. Independent recombination provides the largest gain. C2C substantially improves the shared baseline but adds little once components are already trained independently.

Dataset	$k$ (heads)	Seen	Unseen	HM
HAViD	$k=1$ (atomic)	54.7	0.0	0.0
	$k=2$ ( $v,m$ )	51.1	7.9	13.7
	$k=3$ ( $+t$ )	50.1	7.3	12.7
	$k=4$ ( $+t,l$ )	54.5	5.0	9.1
MECCANO	$k=1$ (atomic)	48.9	0.0	0.0
MECCANO	$k=2$ ( $v,n$ )	48.4	1.7	3.3

Table 5: Finer factorization does not improve compositional generalization. The standard two-component decomposition (

k=2

) provides the best unseen performance. Increasing granularity mainly improves seen accuracy while degrading transfer to unseen compositions.

Figure 6: Remaining headroom is largely a ranking problem. Unseen action top-

K

under the same scoring function used in Table 2 . Atomic prediction remains at zero for all

K

, whereas decomposition frequently ranks the correct unseen composition within its top candidates despite low top-1 accuracy.

Compositional Generalization

Compositional generalization is the ability to combine learned parts and rules in ways that transfer to unseen combinations.

See all 113 papers

Jan 22, 2026

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.

Geo Ahn, Inwoong Lee, Taeoh Kim +3

1Kyung Hee University · ∗This work was done during an internship at NAVER Cloud. · 2NAVER Cloud

Zero-Shot Learning Video Action Recognition Shortcut Learning Compositional Representation Learning Compositional Generalization Zero-Shot Action Recognition

Ask PDF

Jul 12, 2026

Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.

Hao Zheng, Jinyi Huang, Tiantian Zheng +2

Department of Computer Engineering, New York University Abu Dhabi, UAE · Department of Mechanical and Mechatronics Engineering, The University of Auckland, New Zealand · Center for Quantum and Topological Systems, New York University Abu Dhabi, UAE +1

VLM Adaptation Video Action Recognition Parameter-Efficient Fine-Tuning

Ask PDF

Jul 17, 2026

AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning

Vision-Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization when familiar sub-tasks are recombined in unseen configurations. We identify two mutually reinforcing failure modes: trajectory overfitting, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and perceptual shortcut, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding. To address both, we introduce AC-VLA, a plug-and-play Action Compositional learning framework comprising two architecture-agnostic components: (i) a compositional learning module that uses an LLM-driven instruction decomposer and a proprioceptive trajectory aligner to generate dense sub-task supervision, followed by mixed training on complete demonstrations and decomposed data to endow the model with compositional generalization; and (ii) a state-conditioned asymmetric masking strategy that suppresses wrist-view inputs during closed-gripper phases, enforcing global semantic grounding. All components are architectural modification-free and directly integrable into any VLA backbone. Instantiated on

π_{0.5}

and evaluated on LIBERO and LIBERO-OOD benchmarks, AC-VLA achieves a ~28% absolute improvement on compositional OOD tasks while maintaining near-perfect in-distribution performance.

Xiaojiang Peng, Kai Peng, Jie Lu +3

Shenzhen Technology University · Tongji University · Great Bay University +1

Robot Skill Learning Generalization in Robotic Manipulation Robotic Manipulation Vision-Language-Action Models Robustness of VLA Models

Ask PDF

Abstract

Compositional Generalization

Explore similar work

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning