ArtLang: Structured Language-to-Kinematics Grounding for Articulated 3D Actuation

Sylvia Yuan; Dan Wang; Ravi Ramamoorthi; Xinrui Cui

cs.CVAug 15, 2026

ArtLang: Structured Language-to-Kinematics Grounding for Articulated 3D Actuation

Authors: Sylvia Yuan, Dan Wang, Ravi Ramamoorthi, Xinrui Cui

Abstract

Articulated-object reconstructions recover explicit geometry and kinematics, but their parts often remain semantically anonymous and must be controlled through part indices and numerical joint parameters. We present ArtLang, a framework for open-vocabulary language control of persistent reconstructed articulated assets. ArtLang represents an asset as a semantic-kinematic articulation graph and augments its surface with language features and graph-constrained motion. Open-vocabulary proposals are bound to reconstructed parts while allowing uncertain parts to remain unnamed. A typed parser converts a command into a directive graph containing referring expressions, actions, magnitudes, reference frames, and relations. We then solve a global graph-to-graph grounding problem that jointly reasons about semantic, spatial, relational, and kinematic compatibility, with support for null assignments and abstention under ambiguity. Accepted directives are converted into continuous joint targets within the observed motion range and executed through forward kinematics. Experiments on synthetic reconstructions, mesh-based assets, and real captures demonstrate reliable language grounding and continuous articulated control across repeated parts, spatial references, relational commands, and ambiguous instructions.

Figures & tables

Figure 1: Motivation. Reconstructed articulated assets lack a language-addressable interface. ArtLang resolves semantic, spatial, and motion ambiguities to map commands to persistent parts and executable joint coordinates.

Figure 2: Overview of ArtLang. (A) A part-aware reconstruction is represented by an articulation graph and factorized semantic-kinematic field. (B) Open-vocabulary features and proposals are bound to persistent parts by semantic-geometric partial matching. (C) A directive graph is globally grounded using semantic, spatial, relational, and action compatibility. (D) Accepted directives are converted to joint targets and executed by forward kinematics. The resulting graph-to-graph pipeline converts compositional language into globally consistent and executable articulation.

Symbol	Value
$\beta$ (anchor / dense-field mix)	$0.6$
$\tau_{f}$ (dense-field pooling temp.)	$0.1$
$\tau_{s}$ (semantic soft-max temp.)	$0.1$
$w_{\text{ent}}$ (null-score entropy weight)	$0.25$
$w_{\text{conf}}$ (null-score parser-confidence weight)	$1.0$
$\tau_{\text{dummy}}$ (private-dummy cost)	$0.5$

Table 1: Fixed scoring, abstention, and binding hyperparameters. The learned energy weights (

\lambda

g_{r}

b_{\text{null}}

) and the calibrated abstention margin (

\gamma

) are reported in the text.

	Ours	NaiveCLIP	NoSpatial	NaiveWarp
Axis Ang $\downarrow$	23.79	–	–	26.06
Axis Pos $\downarrow$	0.08	–	–	1.03
Part Motion $\downarrow$	6.24	–	–	13.76
CD-m $\downarrow$	26.78	–	–	89.42
L-M Alignment $\downarrow$	70.42	392.52	89.32	127.38
L-Sem Alignment $\uparrow$	0.78	0.33	–	0.71

Table 2: Mesh-input ablation. Each variant removes one language competency while geometry is fixed. Best , Second .

Figure 3: Language grounding ablation. NaiveCLIP [ 41 ] weakens semantic assignment, NoSpatial removes spatial disambiguation, and NaiveWarp replaces per-part articulated motion with a centroid-based warp. Same-colored movable parts indicate those selected by each method for the command.

Figure 4: Continuous language-driven actuation. The grounded part moves along its recovered joint as $t$ varies from $-1$ to $1$ , while all other parts remain fixed.

	ArtMesh	Ours	ArtGS	GaussianArt
Axis Ang $\downarrow$	3.48	3.62	10.53	34.45
Axis Pos $\downarrow$	0.03	0.01	0.11	0.33
Part Motion $\downarrow$	3.63	1.69	3.50	16.48
CD-s $\downarrow$	18.98	19.24	23.56	24.63
CD-m $\downarrow$	15.01	16.62	151.41	38.43

Table 3: Reconstruction on Articulate-100. ArtLang matches ArtMesh in joint and geometry accuracy, and both outperform Gaussian baselines. Lower is better.

Figure 5: Reconstruction on Articulate-100. Each row shows the ground truth, our reconstruction, and the baselines, together with part segmentation and recovered motion. Full two-state results are provided in Figure 14 .

	OPD	Art-Anything	Ours-Mesh	Ours-MV	Ours-SV
Axis Ang $\downarrow$	34.88	26.57	6.98	11.29	23.79
Axis Pos $\downarrow$	4.53	2.08	0.08	0.09	0.08
Part Motion $\downarrow$	25.61	17.81	4.25	4.89	6.24
CD-m $\downarrow$	240.77	114.92	10.26	14.69	26.78
L-M Alignment $\downarrow$	–	–	50.71	57.52	70.42
L-Sem Alignment $\uparrow$	–	–	0.86	0.81	0.78

Table 4: Mesh-input results on Articulate-100. We compare three ArtLang input regimes with OPD and Articulate-Anything. Errors are lower-is-better; semantic and spatial overlaps are higher-is-better.

Figure 6: Mesh-input articulation results. Columns show the ground truth, our three motion-observation settings, Articulate-Anything, and OPD.

Figure 7: Real SplArt captures. The language handle grounds and actuates parts despite noisy geometry and ambiguous boundaries. SplArt provides neither ground-truth meshes nor the original capture lighting, so rendered appearance may differ from the reference images.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Split	#Obj	#Cmd	#Nodes	% null	% rep-part
Train	70	6576	10770	3.9	64.6
Val	15	1362	2220	4.1	67.2
Test	15	1152	1836	4.9	62.3
Total	100	9090	14826	4.0	64.7

Appendix

Table 5: Command-grounding dataset statistics by split.

Method	All $\uparrow$	Multi-directive $\uparrow$	Relational $\uparrow$	Hard $\uparrow$
Global	0.842	0.899	0.827	0.790
Independent	0.783	0.812	0.708	0.543
NoRelation	0.789	0.812	0.701	0.530

Appendix

Table 6: Directive-to-part assignment accuracy: global graph assignment vs. independent argmax and a NoRelation ablation, by subset, on the synthetic command dataset.

Parser	All $\uparrow$	Multi-directive $\uparrow$
Predicted (rule parser)	0.870	0.909
Oracle	0.863	0.901

Appendix

Table 7: Grounding accuracy with the rule-based parser vs. oracle directive graphs, isolating parser error.

Stage	Accuracy $\uparrow$	$n$
Directive-graph parse	0.844	1152
Entity-to-part	0.913	1686
Verb	0.889	1566
Magnitude	1.000	152
Complete-command	0.673	1092
Correct abstention (ambiguous)	0.200	60

Appendix

Table 8: Per-stage accuracy breakdown of the command pipeline on the command dataset (test split, trained resolver

\theta

). Parse (count agreement) : fraction of commands whose rule parse has the same actionable-directive and relation-edge counts as the oracle graph (a necessary condition, not a full labeled-graph match, so it upper-bounds true graph recovery). Entity-to-part : fraction of every directive grounded to a real ground-truth part (all real-part directives, including the reference-only anchor entities the resolver must also place; only null and ambiguous directives are excluded) that the global resolver grounds correctly (

Y_{i}^{\star}=Y_{i,gt}

), on the oracle parse. Verb : fraction of actionable (verb-bearing) directives whose parser-recovered verb (canonicalised to open/close) matches ground truth; its

n

is smaller than the Entity-to-part

n

by exactly the reference-only anchors, which carry a part but no verb. Magnitude : fraction of typed-magnitude probes whose recovered magnitude type is correct over four classes (endpoint, absolute-metric, relative-metric, and fraction; the absolute- and relative-fraction requests are merged into one fraction class, and only the type is scored, not the numeric amount). Complete-command : fraction of non-ambiguous commands the end-to-end pipeline (predicted parse

+

grounding) gets entirely right. Correct abstention : fraction of ambiguous bare-noun directives (

Y_{gt}=-2

) on which the resolver correctly abstains.

n

is the number of scored instances.

Metric	Mean $\uparrow$	Std	95% CI
Directive-graph parse	0.779	0.098	[0.725, 0.834]
Entity-to-part	0.958	0.081	[0.913, 1.000]
Verb	0.805	0.112	[0.743, 0.867]
Magnitude	1.000	0.000	[1.000, 1.000]
Complete-command	0.554	0.184	[0.452, 0.656]
Correct abstention (ambiguous)	0.222	0.441	[0.000, 0.561]

Appendix

Table 9: Per-object dispersion of the command-pipeline metrics across the test objects; because the object-disjoint test split is small we quantify uncertainty. We report, for each metric, the macro mean

\pm

standard deviation over the per-object accuracies and a 95% Student’s-

t

confidence interval, where

n

is the number of test objects that contain that metric’s instances and varies by row; the intervals are truncated to

[0,1]

for display. Metrics are computed exactly as in Table 8 .

Metric	Value
Acc (no abstain)	0.642
Acc (with abstain)	0.682
$\Delta$ (abstention gain)	0.040
Should-abstain fraction	0.300

Appendix

Table 10: Abstention accuracy on an ambiguity stress subset: every should-abstain directive (ambiguous or absent-referent) plus a random, seed-fixed sample of resolvable directives, so should-abstain is 30% of the 500 scored directives (the full command distribution is only a few percent should-abstain, which bounds the achievable gain). The global resolver is held fixed and only the abstention gate is toggled: Acc (no abstain) forces the global assignment to always execute, Acc (with abstain) adds the null/margin gate, and

\Delta

is the abstention gain.

Figure 8: Risk-coverage curve for structured-margin abstention. Abstaining on the lowest-margin directives lowers coverage and reduces selective risk.

Figure 9: Grounding accuracy as the semantic energy is mixed between field-only (

\beta{=}0

) and anchor-only (

\beta{=}1

); a broad combined region improves over either endpoint.

Semantic score	Anchor-decisive $\uparrow$	Field-decisive $\uparrow$	All $\uparrow$
Anchor-only ( $\beta{=}1$ )	1.000	0.520	0.760
Field-only ( $\beta{=}0$ )	0.000	1.000	0.500
Both ( $\beta{=}0.6$ )	1.000	1.000	1.000

Appendix

Table 11: Anchor / field / anchor+field ablation of the semantic score. Anchor-only (

\beta{=}1

) cannot separate same-noun siblings; field-only (

\beta{=}0

) is misled by a distractor part whose dense field spuriously matches the query; a broad range of mixtures around the default (

\beta{=}0.6

) handles both, so the distilled dense field earns its place. These are diagnostic probes over constructed anchor- and field-decisive cases, not the full metric set.

Namer	L-Sem $\uparrow$	L-Spat $\uparrow$	L-M $\downarrow$
Centroid	0.775	0.607	70.42
IoU	0.797	0.547	86.32
Feat	0.784	0.095	326.32
Fusion	0.725	0.413	209.00

Appendix

Table 12: Part-binding cost ablation: language metrics under centroid, mask-IoU, feature-similarity, and learned (fusion-head) matching. The underlying geometry and joint trajectories are fixed across namers, so pure reconstruction metrics are omitted; L-M is retained because naming changes the correspondence used by the language-motion metric.

Method	L-Sem $\uparrow$	L-Spat $\uparrow$
Region-CLIP	0.335	0.084
Grounded-SAM	0.538	0.180
ArtLang (Ours)	0.775	0.607

Appendix

Table 13: Language-grounding baselines vs. ArtLang on the semantic and spatial alignment axes.

Figure 10: Part-binding cost ablation, qualitatively. The same geometry is recolored by the centroid, mask-IoU, feature-similarity, and learned namers, with per-part correctness marks.

Figure 11: Naming metrics as the vision-language proposals are perturbed with increasing noise.

Figure 12: Part naming under proposal noise, qualitatively, at increasing perturbation levels.

Figure 13: Language-grounding baselines, qualitatively. The same geometry is recolored by each baseline’s predicted naming, with per-part correctness marks.

Field	Accuracy $\uparrow$
Action (verb class)	0.864
Referent (part + qualifier)	0.947
Magnitude type	0.959
Intent (action $+$ referent)	0.835

Appendix

Table 14: Intent preservation on free-form paraphrases. Language-model paraphrases of the templated commands (with reordering, anaphora, and out-of-template vocabulary) are read by the typed parser and scored against the intent of the original command by whether the resulting directive would drive the same part the same way: Action is the verb class, Referent requires every spatial qualifier to be preserved and the head noun to match the intended part by string or embedding synonymy (as the semantic anchor grounds it), Magnitude is the request type, and Intent is Action and Referent together.

n

is the number of paraphrases scored.

Stage	Time (ms)
Parse	0.044
Ground	3.979
Actuate	0.009

Appendix

Table 15: Per-stage runtime of the language-to-actuation pipeline, mean over commands (parser, structured grounding, actuation; CPU). Reconstruction and per-asset feature distillation are one-time offline costs incurred before any command is issued and are excluded from this online timing.

Metric	Value
Original-order accuracy $\uparrow$	1.000
Prediction flip rate (states swapped)	1.000
Mean margin (swap-invariant)	0.462

Appendix

Table 16: Endpoint (open/closed) classifier permutation-equivariance. Protocol: the identity of the physical open state is held fixed while its slot changes under the swap, so its correct index flips; the two observed reconstructed states are exchanged and the reported quantity is the prediction flip rate , the fraction of parts whose returned open-state index changes, not an accuracy. A high forward accuracy together with a flip rate of

1.0

is the intended result: the classifier is permutation-equivariant, returning the index of the same appearance-based choice under either input ordering (a genuinely order-driven predictor that always picks a fixed slot would instead have a flip rate of

0

). This is an architectural consistency check, not evidence of learned open/closed semantics. The margin is invariant to the swap.

Figure 14: Reconstruction comparison on Articulate-100, both states, for a representative subset of objects. Each row is an object; the two state blocks show the rest and articulated states, with columns for the ground truth, our reconstruction, and the Gaussian baselines, including the per-part segmentation and the joint motion overlay.

Figure 15: Mesh-input comparison on Articulate-100, both states, for a representative subset of objects. Columns show the ground truth and our mesh-input, multi-view, and single-view variants next to Articulate-Anything [ 24 ] and OPD [ 19 ] , rendered with texture in both the rest and articulated states.

Figure 16: Ablation comparison, both states, for a representative subset of objects. Columns show the ground truth, our model, and the NaiveCLIP, NoSpatial, and NaiveWarp ablations on the same given mesh, in the rest and articulated states.

Figure 17: Full continuous actuation sweep. Every movable part of the object is driven along its learned trajectory in turn, one row per part, while the remaining parts are held at rest, sweeping the signed linear factor $s$ of Eq. ( 25 ) from $-1$ to $1$ : $s\in[0,1]$ reproduces the observed state $0\!\to\!1$ configurations (for this displayed example state $0$ happens to be the more-open pose, so left-to-right reads as open-to-close; the endpoint classifier does not assume or use this ordering and decides which state is open from appearance), and $s<0$ extrapolates the learned joint past the observed rest state. The one-row version in Figure 4 shows a single part of this figure.

Explore similar work

Jun 12, 2026cs.CV

Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control

Reconstructing articulated 3D objects is important for animation, gaming, and robotic simulations. Recent neural networks can estimate the articulated structure of 3D objects, but their generalization remains limited by the scarcity of annotated data for this task. To address this gap, we introduce Instruct-Particulate, a model that takes a 3D mesh together with a target kinematic specification, including part descriptions, connectivity, joint types, and optional point prompts, and predicts the corresponding kinematic part segmentation and joint motion parameters. The kinematic specification disambiguates the task and allows the model to target annotations of different granularity, thereby making it possible to use more abundant heterogeneous training data. At test time, the kinematic specification can be obtained automatically from large-scale vision-language models, so the model can be applied to any input mesh. To train our model at scale, we construct a heterogeneous dataset of more than 150,000 articulated 3D objects, extending existing publicly available collections with data obtained by partially labelling other 3D models (monolithic or already decomposed into parts) with kinematic labels by means of vision-language models. Experiments show that our model generalizes better across categories and to AI-generated meshes, enabling articulated asset reconstruction from real-world images via image-to-3D models.

Ruining Li, Yuxin Yao, Matt Zhou +5

University of Oxford · University of Cambridge · 3Nanyang Technological University

Articulated Object Reconstruction 3D Part Segmentation 3D Reconstruction Articulated Object Manipulation

Ask PDF

May 14, 2026cs.CV

Articraft: An Agentic System for Scalable Articulated 3D Asset Generation

A bottleneck in learning to understand articulated 3D objects is the lack of large and diverse datasets. In this paper, we propose to leverage large language models (LLMs) to close this gap and generate articulated assets at scale. We reduce the problem of generating an articulated 3D asset to that of writing a program that builds it. We then introduce a new agentic system, Articraft, that writes such programs automatically. We design a programmatic interface and harness to help the LLM do so effectively. The LLM writes code against a domain-specific SDK for defining parts, composing geometry, specifying joints, and writing tests to validate the resulting assets. The harness exposes a restricted workspace and interface to the LLM, validates the resulting assets, and returns structured feedback. In this way, the LLM is not distracted by details such as authoring a URDF file or managing a complex software environment. We show that this produces higher-quality assets than both state-of-the-art articulated-asset generators and general-purpose coding agents. Using Articraft, we build Articraft-10K, a curated dataset of over 10K articulated assets spanning 245 categories, and show its utility both for training models of articulated assets and in downstream applications such as robotics simulation and virtual reality.

Matt Zhou, Ruining Li, Xiaoyang Lyu +6

University of Cambridge · University of Oxford · Nanyang Technological University

3D Asset Generation LLM-Based Program Synthesis Synthetic Data Generation for Robotics

Ask PDF

Date pendingcs.CV

Artic-O: End-to-End Articulated Object Reconstruction via Latent Geometry Learning

Reconstructing articulated objects from sparse images requires recovering complete geometry, movable parts, and motion parameters. Recent methods typically separate geometry reconstruction, part reasoning, and articulation estimation into different stages. This separation can weaken consistency between shape, active parts, and motion, while also incurring substantial inference cost. We introduce Artic-O, an end-to-end, feed-forward framework for articulated object reconstruction via latent geometry learning. Instead of fitting geometry in image or view space, Artic-O maps sparse multi-state observations into a pretrained latent geometry space, where a frozen flow-matching decoder provides a complete-shape prior for recovering visible and occluded structures. To connect geometry with articulation, Artic-O fuses visual tokens, geometry latents, and point-wise decoder features in an image-grounded part-reasoning module for active-part segmentation and articulation prediction. We further train the model with a geometry-to-articulation curriculum and a decoupled two-pass strategy to balance reconstruction and part-level supervision. On PartNet-Mobility, Artic-O achieves strong reconstruction quality while being substantially more efficient than LARM, a strong prior method. It reduces Chamfer Distance, improves F-score, and achieves comparable or better articulation accuracy across most joint metrics, while reducing inference time from 9 minutes to about 0.3 seconds per object.

Xuyang Wang, Zhenyu Li, Jian Ding +4

Australian National University, Canberra, Australia · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

Articulated Object Reconstruction 3D Reconstruction Sparse-View 3D Reconstruction Feed-Forward 3D Reconstruction

Ask PDF