Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognition

Changyi Li; Yu Xiao

cs.CVSep 4, 2026

Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognition

Authors: Changyi Li, Yu Xiao

Organizations: Department of Information and Communications Engineering Aalto University 02150 Espoo, Finland

Abstract

Assembly actions are compositional: they combine a manipulation with a part or tool. In deployment, systems routinely encounter novel combinations of familiar components, yet an atomic action classifier assigns every unseen combination exactly zero probability by construction. The prevailing solution is verb--noun decomposition, which predicts components separately and recombines them to reach unseen actions. While widely adopted, how decomposition generalizes under compositional shift remains poorly understood. We present a systematic analysis of verb--noun decomposition across three assembly datasets (MECCANO, HAViD, and IMPACT). Although decomposition escapes the atomic ceiling, its generalization extends only partially beyond it. Unseen-composition performance remains strongly tied to the co-occurrence structure of the training data, indicating that much of the observed gain arises from interpolation within densely supported regions of the compositional space rather than from unconstrained recombination. Across datasets, failures consistently concentrate on the larger-vocabulary component, and IMPACT's verb-heavy vocabulary reverses the bottleneck from nouns to verbs. We further show that shared-encoder training introduces component entanglement, encouraging reliance on co-occurrence patterns that transfer poorly to unseen compositions and trailing independent recombination by up to $6.0\times$ in harmonic mean. Taken together, these findings explain why decomposition achieves only partial compositional generalization in practice. By identifying primitive support, vocabulary asymmetry, and component entanglement as connected sources of error, we provide a portable diagnostic framework for studying compositional recognition beyond aggregate accuracy. Code: https://github.com/hisalaheiyo/assembly.

Figures & tables

Figure 1: Recognition formulations studied in this paper. Top: atomic prediction ; middle: shared decomposition ; bottom: independent decomposition . Atomic prediction treats each action as a single class, while the decomposition variants predict verbs and nouns separately using either a shared or independent backbone and combine them via $\log P(v)+\log P(n)$ . The coloured output boxes indicate the reachable output space: atomic prediction is limited to training actions, whereas decomposition spans the full verb $\times$ noun space and can represent unseen compositions.

Dataset	$\|\mathcal{V}\|$	$\|\mathcal{N}\|$	grid	obs. (sparsity)	larger vocab	viewpoints / hands
MECCANO	12	23	276	61 (22.1%)	noun ( $1.9\times$ )	egocentric, single-hand
HAViD	11	29	319	165 (51.7%)	object ( $2.6\times$ )	3 static, bimanual
IMPACT	22	18	396	136 (34.3%)	verb ( $1.2\times$ )	5 views (incl. ego), bimanual

Table 1: Structural diversity of the datasets used in this study. IMPACT uniquely reverses the common noun-heavy regime, enabling a direct test of whether the bottleneck follows vocabulary size rather than component type.

Dataset	Config	Seen	Unseen	HM
MECCANO	Atomic	48.9	0.0	0.0
	Shared	48.4	1.7	3.3
	Independent	44.3	12.8	19.8
	C2C-shared	27.9	13.2	17.9
	C2C-indep	46.6	16.5	24.3
HAViD	Atomic	54.7	0.0	0.0

Table 2: Performance on the compositional split. Seen and unseen action top-1 accuracy and harmonic mean (HM) for five recognition formulations across three datasets. Atomic prediction serves as the unreachable baseline, while shared and independent decomposition make unseen compositions reachable. This table provides the reference point for all subsequent analyses.

Figure 2: Unseen accuracy increasingly follows primitive support as the composition grid becomes denser. Each point is an unseen verb–noun pair. Correlation strength rises from MECCANO to IMPACT to HAViD, mirroring the density of the observed composition grid. High-support pairs achieve the strongest performance, whereas low-support pairs remain near zero.

Figure 3: The bottleneck follows vocabulary size rather than component identity. (a) Unseen errors partitioned by component. Failures consistently concentrate on the larger-vocabulary component. (b) The verb bottleneck in IMPACT persists across all five views, indicating that the asymmetry follows vocabulary structure rather than viewpoint.

Dataset	$\|\mathcal{V}\|$	$\|\mathcal{N}\|$	both	obj err ( $\pi_{v}$ )	verb err ( $\pi_{n}$ )	both wrong	bottleneck
MECCANO	12	23	12.8	33.5	26.2	27.5	noun/obj
HAViD	11	29	18.6	39.6	12.9	28.9	noun/obj
IMPACT	22	18	22.1	15.1	32.6	30.2	verb
IMPACT per view (verb bottleneck is view-invariant):
front			24.5	15.1	31.8	28.6	verb
left			25.1	17.1	30.8	27.1	verb

Table 3: The bottleneck follows vocabulary size rather than component identity. Across datasets, unseen failures consistently concentrate on the larger-vocabulary component. IMPACT reverses the common noun-heavy regime, and the bottleneck reverses with it across all five views.

	shortcut $\Delta$ (pp)			grid	seen-margin
Dataset	verb	object	$\rho$ (D2)	density	(v, o) pp
MECCANO	6.4	11.5	0.49	22.1%	(-0.3, -0.6)
HAViD	5.9	8.7	0.69	51.7%	(-1.2, -1.9)
IMPACT	12.5	14.5	0.56	34.3%	(+1.8, +0.7)
$\Delta=$ excess seen $\to$ unseen drop of joint over solo (shortcut). seen-margin $\approx 0$ rules out capacity.

Table 4: Evidence for component entanglement in shared training. The shortcut score

\Delta

measures the excess seen-to-unseen drop of a jointly trained component head relative to its independently trained counterpart. Large

\Delta

values together with near-zero seen margins indicate that shared training provides little benefit on seen compositions while substantially increasing the unseen generalization gap.

Figure 4: More factorization does not imply better compositional generalization. HAViD performance under

k=1\ldots 4

factorization levels. The standard two-component decomposition (

k=2

) achieves the best unseen performance. Further factorization recovers seen accuracy but degrades generalization. Inset: the corresponding

k{=}1\!\to\!2

transition on MECCANO and IMPACT.

Figure 5: Interventions change performance levels but not the bottleneck. Unseen HM for shared and independent formulations, with and without C2C. Independent recombination provides the largest gain. C2C substantially improves the shared baseline but adds little once components are already trained independently.

Dataset	$k$ (heads)	Seen	Unseen	HM
HAViD	$k=1$ (atomic)	54.7	0.0	0.0
	$k=2$ ( $v,m$ )	51.1	7.9	13.7
	$k=3$ ( $+t$ )	50.1	7.3	12.7
	$k=4$ ( $+t,l$ )	54.5	5.0	9.1
MECCANO	$k=1$ (atomic)	48.9	0.0	0.0
MECCANO	$k=2$ ( $v,n$ )	48.4	1.7	3.3

Table 5: Finer factorization does not improve compositional generalization. The standard two-component decomposition (

k=2

) provides the best unseen performance. Increasing granularity mainly improves seen accuracy while degrading transfer to unseen compositions.

Figure 6: Remaining headroom is largely a ranking problem. Unseen action top-

K

under the same scoring function used in Table 2 . Atomic prediction remains at zero for all

K

, whereas decomposition frequently ranks the correct unseen composition within its top candidates despite low top-1 accuracy.

Compositional Generalization

Compositional generalization is the ability to combine learned parts and rules in ways that transfer to unseen combinations.

See all 113 papers

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition
Jan 22, 2026Geo Ahn, Inwoong Lee, Taeoh Kim +3Zero-Shot Learning Video Action Recognition
1Kyung Hee University · ∗This work was done during an internship at NAVER Cloud. · 2NAVER Cloud
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
Jul 12, 2026Hao Zheng, Jinyi Huang, Tiantian Zheng +2VLM Adaptation Video Action Recognition
Department of Computer Engineering, New York University Abu Dhabi, UAE · Department of Mechanical and Mechatronics Engineering, The University of Auckland, New Zealand · Center for Quantum and Topological Systems, New York University Abu Dhabi, UAE +1
AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning
Jul 17, 2026Xiaojiang Peng, Kai Peng, Jie Lu +3Robot Skill Learning Generalization in Robotic Manipulation
Shenzhen Technology University · Tongji University · Great Bay University +1

Jan 22, 2026J/K move · Enter open · S save

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim+2

1Kyung Hee University · ∗This work was done during an internship at NAVER Cloud. · 2NAVER Cloud

Zero-Shot Learning Video Action Recognition Shortcut Learning Compositional Representation Learning Compositional Generalization Zero-Shot Action Recognition

Open paper AskPDF arXiv

Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.

Abstract

Compositional Generalization

Explore similar work

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning