cs.CVAug 15, 2026
SaveArtLang: Structured Language-to-Kinematics Grounding for Articulated 3D Actuation
Organizations: University of California San Diego · University of North Texas
Abstract
Articulated-object reconstructions recover explicit geometry and kinematics, but their parts often remain semantically anonymous and must be controlled through part indices and numerical joint parameters. We present ArtLang, a framework for open-vocabulary language control of persistent reconstructed articulated assets. ArtLang represents an asset as a semantic-kinematic articulation graph and augments its surface with language features and graph-constrained motion. Open-vocabulary proposals are bound to reconstructed parts while allowing uncertain parts to remain unnamed. A typed parser converts a command into a directive graph containing referring expressions, actions, magnitudes, reference frames, and relations. We then solve a global graph-to-graph grounding problem that jointly reasons about semantic, spatial, relational, and kinematic compatibility, with support for null assignments and abstention under ambiguity. Accepted directives are converted into continuous joint targets within the observed motion range and executed through forward kinematics. Experiments on synthetic reconstructions, mesh-based assets, and real captures demonstrate reliable language grounding and continuous articulated control across repeated parts, spatial references, relational commands, and ambiguous instructions.
Figures & tables
Figure 1: Motivation. Reconstructed articulated assets lack a language-addressable interface. ArtLang resolves semantic, spatial, and motion ambiguities to map commands to persistent parts and executable joint coordinates.
Figure 2: Overview of ArtLang. (A) A part-aware reconstruction is represented by an articulation graph and factorized semantic-kinematic field. (B) Open-vocabulary features and proposals are bound to persistent parts by semantic-geometric partial matching. (C) A directive graph is globally grounded using semantic, spatial, relational, and action compatibility. (D) Accepted directives are converted to joint targets and executed by forward kinematics. The resulting graph-to-graph pipeline converts compositional language into globally consistent and executable articulation.
| Symbol | Value |
| (anchor / dense-field mix) | |
| (dense-field pooling temp.) | |
| (semantic soft-max temp.) | |
| (null-score entropy weight) | |
| (null-score parser-confidence weight) | |
| (private-dummy cost) |
Table 1: Fixed scoring, abstention, and binding hyperparameters. The learned energy weights ( , , ) and the calibrated abstention margin ( ) are reported in the text.
| Ours | NaiveCLIP | NoSpatial | NaiveWarp | |
| Axis Ang | 23.79 | – | – | 26.06 |
| Axis Pos | 0.08 | – | – | 1.03 |
| Part Motion | 6.24 | – | – | 13.76 |
| CD-m | 26.78 | – | – | 89.42 |
| L-M Alignment | 70.42 | 392.52 | 89.32 | 127.38 |
| L-Sem Alignment | 0.78 | 0.33 | – | 0.71 |
Table 2: Mesh-input ablation. Each variant removes one language competency while geometry is fixed. Best , Second .
Figure 3: Language grounding ablation. NaiveCLIP [ 41 ] weakens semantic assignment, NoSpatial removes spatial disambiguation, and NaiveWarp replaces per-part articulated motion with a centroid-based warp. Same-colored movable parts indicate those selected by each method for the command.
Figure 4: Continuous language-driven actuation. The grounded part moves along its recovered joint as varies from to , while all other parts remain fixed.
| ArtMesh | Ours | ArtGS | GaussianArt | |
| Axis Ang | 3.48 | 3.62 | 10.53 | 34.45 |
| Axis Pos | 0.03 | 0.01 | 0.11 | 0.33 |
| Part Motion | 3.63 | 1.69 | 3.50 | 16.48 |
| CD-s | 18.98 | 19.24 | 23.56 | 24.63 |
| CD-m | 15.01 | 16.62 | 151.41 | 38.43 |
Table 3: Reconstruction on Articulate-100. ArtLang matches ArtMesh in joint and geometry accuracy, and both outperform Gaussian baselines. Lower is better.
Figure 5: Reconstruction on Articulate-100. Each row shows the ground truth, our reconstruction, and the baselines, together with part segmentation and recovered motion. Full two-state results are provided in Figure 14 .
| OPD | Art-Anything | Ours-Mesh | Ours-MV | Ours-SV | |
| Axis Ang | 34.88 | 26.57 | 6.98 | 11.29 | 23.79 |
| Axis Pos | 4.53 | 2.08 | 0.08 | 0.09 | 0.08 |
| Part Motion | 25.61 | 17.81 | 4.25 | 4.89 | 6.24 |
| CD-m | 240.77 | 114.92 | 10.26 | 14.69 | 26.78 |
| L-M Alignment | – | – | 50.71 | 57.52 | 70.42 |
| L-Sem Alignment | – | – | 0.86 | 0.81 | 0.78 |
Table 4: Mesh-input results on Articulate-100. We compare three ArtLang input regimes with OPD and Articulate-Anything. Errors are lower-is-better; semantic and spatial overlaps are higher-is-better.
Figure 6: Mesh-input articulation results. Columns show the ground truth, our three motion-observation settings, Articulate-Anything, and OPD.
Figure 7: Real SplArt captures. The language handle grounds and actuates parts despite noisy geometry and ambiguous boundaries. SplArt provides neither ground-truth meshes nor the original capture lighting, so rendered appearance may differ from the reference images.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Split | #Obj | #Cmd | #Nodes | % null | % rep-part |
| Train | 70 | 6576 | 10770 | 3.9 | 64.6 |
| Val | 15 | 1362 | 2220 | 4.1 | 67.2 |
| Test | 15 | 1152 | 1836 | 4.9 | 62.3 |
| Total | 100 | 9090 | 14826 | 4.0 | 64.7 |
Table 5: Command-grounding dataset statistics by split.
| Method | All | Multi-directive | Relational | Hard |
| Global | 0.842 | 0.899 | 0.827 | 0.790 |
| Independent | 0.783 | 0.812 | 0.708 | 0.543 |
| NoRelation | 0.789 | 0.812 | 0.701 | 0.530 |
Table 6: Directive-to-part assignment accuracy: global graph assignment vs. independent argmax and a NoRelation ablation, by subset, on the synthetic command dataset.
| Parser | All | Multi-directive |
| Predicted (rule parser) | 0.870 | 0.909 |
| Oracle | 0.863 | 0.901 |
Table 7: Grounding accuracy with the rule-based parser vs. oracle directive graphs, isolating parser error.
| Stage | Accuracy | |
| Directive-graph parse | 0.844 | 1152 |
| Entity-to-part | 0.913 | 1686 |
| Verb | 0.889 | 1566 |
| Magnitude | 1.000 | 152 |
| Complete-command | 0.673 | 1092 |
| Correct abstention (ambiguous) | 0.200 | 60 |
Table 8: Per-stage accuracy breakdown of the command pipeline on the command dataset (test split, trained resolver ). Parse (count agreement) : fraction of commands whose rule parse has the same actionable-directive and relation-edge counts as the oracle graph (a necessary condition, not a full labeled-graph match, so it upper-bounds true graph recovery). Entity-to-part : fraction of every directive grounded to a real ground-truth part (all real-part directives, including the reference-only anchor entities the resolver must also place; only null and ambiguous directives are excluded) that the global resolver grounds correctly ( ), on the oracle parse. Verb : fraction of actionable (verb-bearing) directives whose parser-recovered verb (canonicalised to open/close) matches ground truth; its is smaller than the Entity-to-part by exactly the reference-only anchors, which carry a part but no verb. Magnitude : fraction of typed-magnitude probes whose recovered magnitude type is correct over four classes (endpoint, absolute-metric, relative-metric, and fraction; the absolute- and relative-fraction requests are merged into one fraction class, and only the type is scored, not the numeric amount). Complete-command : fraction of non-ambiguous commands the end-to-end pipeline (predicted parse grounding) gets entirely right. Correct abstention : fraction of ambiguous bare-noun directives ( ) on which the resolver correctly abstains. is the number of scored instances.
| Metric | Mean | Std | 95% CI |
| Directive-graph parse | 0.779 | 0.098 | [0.725, 0.834] |
| Entity-to-part | 0.958 | 0.081 | [0.913, 1.000] |
| Verb | 0.805 | 0.112 | [0.743, 0.867] |
| Magnitude | 1.000 | 0.000 | [1.000, 1.000] |
| Complete-command | 0.554 | 0.184 | [0.452, 0.656] |
| Correct abstention (ambiguous) | 0.222 | 0.441 | [0.000, 0.561] |
Table 9: Per-object dispersion of the command-pipeline metrics across the test objects; because the object-disjoint test split is small we quantify uncertainty. We report, for each metric, the macro mean standard deviation over the per-object accuracies and a 95% Student’s- confidence interval, where is the number of test objects that contain that metric’s instances and varies by row; the intervals are truncated to for display. Metrics are computed exactly as in Table 8 .
| Metric | Value |
| Acc (no abstain) | 0.642 |
| Acc (with abstain) | 0.682 |
| (abstention gain) | 0.040 |
| Should-abstain fraction | 0.300 |
Table 10: Abstention accuracy on an ambiguity stress subset: every should-abstain directive (ambiguous or absent-referent) plus a random, seed-fixed sample of resolvable directives, so should-abstain is 30% of the 500 scored directives (the full command distribution is only a few percent should-abstain, which bounds the achievable gain). The global resolver is held fixed and only the abstention gate is toggled: Acc (no abstain) forces the global assignment to always execute, Acc (with abstain) adds the null/margin gate, and is the abstention gain.
Figure 8: Risk-coverage curve for structured-margin abstention. Abstaining on the lowest-margin directives lowers coverage and reduces selective risk.
Figure 9: Grounding accuracy as the semantic energy is mixed between field-only ( ) and anchor-only ( ); a broad combined region improves over either endpoint.
| Semantic score | Anchor-decisive | Field-decisive | All |
| Anchor-only ( ) | 1.000 | 0.520 | 0.760 |
| Field-only ( ) | 0.000 | 1.000 | 0.500 |
| Both ( ) | 1.000 | 1.000 | 1.000 |
Table 11: Anchor / field / anchor+field ablation of the semantic score. Anchor-only ( ) cannot separate same-noun siblings; field-only ( ) is misled by a distractor part whose dense field spuriously matches the query; a broad range of mixtures around the default ( ) handles both, so the distilled dense field earns its place. These are diagnostic probes over constructed anchor- and field-decisive cases, not the full metric set.
| Namer | L-Sem | L-Spat | L-M |
| Centroid | 0.775 | 0.607 | 70.42 |
| IoU | 0.797 | 0.547 | 86.32 |
| Feat | 0.784 | 0.095 | 326.32 |
| Fusion | 0.725 | 0.413 | 209.00 |
Table 12: Part-binding cost ablation: language metrics under centroid, mask-IoU, feature-similarity, and learned (fusion-head) matching. The underlying geometry and joint trajectories are fixed across namers, so pure reconstruction metrics are omitted; L-M is retained because naming changes the correspondence used by the language-motion metric.
| Method | L-Sem | L-Spat |
| Region-CLIP | 0.335 | 0.084 |
| Grounded-SAM | 0.538 | 0.180 |
| ArtLang (Ours) | 0.775 | 0.607 |
Table 13: Language-grounding baselines vs. ArtLang on the semantic and spatial alignment axes.
Figure 10: Part-binding cost ablation, qualitatively. The same geometry is recolored by the centroid, mask-IoU, feature-similarity, and learned namers, with per-part correctness marks.
Figure 11: Naming metrics as the vision-language proposals are perturbed with increasing noise.
Figure 12: Part naming under proposal noise, qualitatively, at increasing perturbation levels.
Figure 13: Language-grounding baselines, qualitatively. The same geometry is recolored by each baseline’s predicted naming, with per-part correctness marks.
| Field | Accuracy |
| Action (verb class) | 0.864 |
| Referent (part + qualifier) | 0.947 |
| Magnitude type | 0.959 |
| Intent (action referent) | 0.835 |
Table 14: Intent preservation on free-form paraphrases. Language-model paraphrases of the templated commands (with reordering, anaphora, and out-of-template vocabulary) are read by the typed parser and scored against the intent of the original command by whether the resulting directive would drive the same part the same way: Action is the verb class, Referent requires every spatial qualifier to be preserved and the head noun to match the intended part by string or embedding synonymy (as the semantic anchor grounds it), Magnitude is the request type, and Intent is Action and Referent together. is the number of paraphrases scored.
| Stage | Time (ms) |
| Parse | 0.044 |
| Ground | 3.979 |
| Actuate | 0.009 |
Table 15: Per-stage runtime of the language-to-actuation pipeline, mean over commands (parser, structured grounding, actuation; CPU). Reconstruction and per-asset feature distillation are one-time offline costs incurred before any command is issued and are excluded from this online timing.
| Metric | Value |
| Original-order accuracy | 1.000 |
| Prediction flip rate (states swapped) | 1.000 |
| Mean margin (swap-invariant) | 0.462 |
Table 16: Endpoint (open/closed) classifier permutation-equivariance. Protocol: the identity of the physical open state is held fixed while its slot changes under the swap, so its correct index flips; the two observed reconstructed states are exchanged and the reported quantity is the prediction flip rate , the fraction of parts whose returned open-state index changes, not an accuracy. A high forward accuracy together with a flip rate of is the intended result: the classifier is permutation-equivariant, returning the index of the same appearance-based choice under either input ordering (a genuinely order-driven predictor that always picks a fixed slot would instead have a flip rate of ). This is an architectural consistency check, not evidence of learned open/closed semantics. The margin is invariant to the swap.
Figure 14: Reconstruction comparison on Articulate-100, both states, for a representative subset of objects. Each row is an object; the two state blocks show the rest and articulated states, with columns for the ground truth, our reconstruction, and the Gaussian baselines, including the per-part segmentation and the joint motion overlay.
Figure 15: Mesh-input comparison on Articulate-100, both states, for a representative subset of objects. Columns show the ground truth and our mesh-input, multi-view, and single-view variants next to Articulate-Anything [ 24 ] and OPD [ 19 ] , rendered with texture in both the rest and articulated states.
Figure 16: Ablation comparison, both states, for a representative subset of objects. Columns show the ground truth, our model, and the NaiveCLIP, NoSpatial, and NaiveWarp ablations on the same given mesh, in the rest and articulated states.
Figure 17: Full continuous actuation sweep. Every movable part of the object is driven along its learned trajectory in turn, one row per part, while the remaining parts are held at rest, sweeping the signed linear factor of Eq. ( 25 ) from to : reproduces the observed state configurations (for this displayed example state happens to be the more-open pose, so left-to-right reads as open-to-close; the endpoint classifier does not assume or use this ordering and decides which state is open from appearance), and extrapolates the learned joint past the observed rest state. The one-row version in Figure 4 shows a single part of this figure.