We introduce avatar referring segmentation, which selects regions of animatable Gaussian avatars from language queries. A query can also specify a pose or motion to select matching intervals. This lets users interact with avatars for local editing and motion inspection. However, making these selections with existing language Gaussian splatting methods poses two challenges: adjacent parts and garments are hard to separate and may be referred to indirectly (region ambiguity), while fine-grained motions can look alike in RGB (motion ambiguity). We address these challenges with HumanSkill-4D, a training-free framework that shares a multimodal large language model (MLLM) and geometric operators across avatars. For each avatar, a semantic memory links body-part and garment categories to persistent Gaussian regions and appearance descriptions. The MLLM uses category names and descriptions to select categories by appearance, relation, or function. To find matching intervals, it composes operators into programs that check query conditions on driving joint trajectories stored in a motion memory. These category choices and operator outputs make selections traceable, while composing or adding operators supports new motion conditions. To evaluate this task, we introduce three human-annotated benchmarks and show that HumanSkill-4D outperforms the baselines in part segmentation and 3D and 4D referring segmentation, with compact memories and only minutes of setup per avatar.
Figures & tables
Figure 1: Avatar referring segmentation with HumanSkill-4D . (a) A 3D query selects the described region of a Gaussian avatar ( Where ). (b) A 4D query also selects the intervals in which the specified pose or motion occurs ( When ). See our project page for video demos.
Figure 2: HumanSkill-4D overview. (Offline) We build a semantic memory for each avatar, a motion memory for its driving pose sequence, and an operator registry shared across avatars. (Online) The frozen MLLM selects target categories from their names and descriptions, and their stored scores determine the spatial mask. For a 4D query, it also reads the registry’s text cards and composes operators into a motion program, and running the program on the joint trajectories gives the temporal mask. The two masks give the spatiotemporal selection. In this example, the program identifies intervals when both upper arms move outward, and StateAtEnd retains each movement interval whose last frame has both arms horizontal (orange band).
Method
Venue
mIoU per avatar
Overall
313
315
377
386
387
390
392
393
394
OA
mAcc
mIoU
4D-DRESS
CVPR’24
35.9
33.9
34.7
24.7
27.5
20.0
37.5
24.9
24.9
78.6
35.7
31.2
OpenHuman3D
3DV’25
47.7
44.8
32.3
35.6
30.9
25.9
44.3
45.9
26.9
66.6
51.8
41.2
OpenHuman4D
BMVC’25
48.0
37.8
40.3
31.8
27.9
26.0
43.6
41.9
29.6
77.6
52.8
40.0
Find3D
ICCV’25
17.0
13.3
13.1
13.4
8.8
8.1
13.5
12.1
10.7
14.9
31.5
12.7
HumanSkill-4D
73.4
77.6
74.5
65.6
52.1
50.4
65.9
67.8
48.0
94.2
79.5
75.0
Table 1: Part segmentation on ZJU-Refer-3D. OA, mAcc, and mIoU (%) over 16 categories.
Method
Part
App.
Rel.
All
LangSplat
57.0
68.0
4.1
56.2
LangSplat †
71.3
72.2
4.8
61.9
ReferSplat
28.6
47.6
5.1
37.4
ReferSplat †
52.1
35.4
12.7
35.3
OpenHuman3D
40.3
54.9
3.1
44.2
Dr. Splat
8.5
16.2
9.1
13.6
Table 2: 3D referring segmentation on ZJU-Refer-3D. mIoU (%) on part, appearance (App.), and relation/function (Rel.) queries; † uses our parser Φparse (i.e., Sapiens2).
Figure 3: Qualitative 3D referring segmentation on ZJU-Refer-3D. Two held-out views and a detail crop per query; selected regions are magenta .
MLLM
mIoU
vIoU
Qwen3.6-27B
82.4
44.3
Gemma 4-31B
55.3
45.0
Mistral 3.2-24B
58.6
38.8
Qwen3.8-27B
82.4
45.0
Table 4: Ablations on MLLM choice ( left ), semantic category selector ( middle ), and score threshold τ ( right ). We report 3D mIoU and 4D vIoU (%).
Figure 4: Qualitative 4D referring segmentation on HuMMan-Refer-4D. GT and predicted regions are magenta and intervals green .
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Measurement type
Definition
State templates (20)
Change templates (26)
Coordinate frame
Joint angle
Angle at a 3-joint hinge (e.g., shoulder–elbow–wrist)
(7) straight, broad bent, and 5 mutually exclusive bent degrees (slight, partial, right angle, deep, full)
(2) bend; straighten
Root-local
Relative position
Point relative to a reference (state) or segment endpoint relative to its start (change), along a fixed-frame axis
Table S1: Operator registry R . Templates grouped by measurement; counts in parentheses.
Figure S1: Running-query templates. Move outward (left) and horizontal (right) on both upper arms.
Figure S2: Operator measurements of the running query. The top panel shows how far each elbow lies outward from its shoulder ( ρ1,ρ2 ), and the bottom panel shows the angle of each upper arm from vertical ( ρ3,ρ4 ). Solid lines are the left arm and dashed lines the right, and amber marks the frames each operator returns.
Relation
Inputs
Output
CoOccurrence
2 or more movements
occurrences formed by intervals shared by every movement
AllStates
2 or more states
frames shared by all states
Sequence
movements in order; optional leading state
span from the first to the last movement, linking each last occurrence to its nearest eligible predecessors with no maximum gap; consecutive movements may share a turning frame; a leading state must end before the first movement starts and does not extend the span
Alternation
2 movements A and B
spans of alternating chains of 3 or more occurrences; consecutive movements have disjoint interiors and may share a turning frame
StateAtEnd
a movement and a state
occurrences whose last frame lies in the state
StateThroughout
a movement and a state
occurrences fully covered by the state
Appendix
Table S2: Temporal relations.
Figure S3: Execution record of the running query. Bars mark the frames returned by each operator and relation; dashed lines and bodies mark the last frame of each co-occurrence.
Card (kind)
Card summary
move outward (change)
Measures: the segment endpoint, relative to its start, moves toward its anatomical outer side. Parts: arm, forearm, leg, lower leg, thigh, upper arm.
horizontal (state)
Measures: the segment is horizontal relative to gravity.
CoOccurrence (relation)
Inputs: two or more movements. Meaning: the movements must hold on the same frames. Returns: every shared portion on which all of the movements hold.
AllStates (relation)
Inputs: two or more states. Meaning: the states must hold on the same frames. Returns: the frames on which every state holds.
StateAtEnd (relation)
Inputs: a movement, then a state. Meaning: the state describes the pose in which the movement ends. Returns: the movement occurrence whose last frame lies inside the state.
Appendix
Table S3: Text cards used for the running query.
Method
Segmentation
Interpretation
Stored information
3D referring segmentation
LangSplat
SAM
CLIP
Features + networks
ReferSplat
Grounded-SAM
BERT + attention
Features + networks
OpenHuman3D
SAM
HumanCLIP
Masks + features
HumanSkill-4D
Sapiens2
MLLM
Scores + text
4D referring segmentation
Appendix
Table S4: Method components for the computational cost comparison in Tab. 5 . Stored information lists component types. Segmentation: 2D mask source; × : none.
Model
mIoU
vIoU
Gemma 4 E2B
60.2
0.0
Ministral 3 8B
63.1
20.4
Qwen3-VL-8B
78.4
16.2
Phi-4
49.9
2.6
Appendix
Table S5: Smaller MLLMs. 3D mIoU and 4D vIoU (%).
Figure S4: Qualitative part segmentation on ZJU-Refer-3D.
Figure S5: Qualitative 3D referring segmentation on ZJU-Refer-3D. † uses Sapiens2 masks.
Figure S6: Qualitative 3D referring segmentation on ZJU-Refer-3D. † uses Sapiens2 masks.
Figure S7: Qualitative 4D referring segmentation on HuMMan-Refer-4D.
Figure S8: Qualitative 4D referring segmentation on ZJU-Refer-4D.
Figure S9: Qualitative 4D referring segmentation on ZJU-Refer-4D.
Figure S10: Qualitative 4D referring segmentation on ZJU-Refer-4D.
ID
Query
Q01
Select both hands.
Q02
Select both arms.
Q03
Select both shoes.
Q04
Select the upper clothing and lower clothing.
Q05
Select the garment with a zipper down the front.
Q06
Select the yellow garment.
Appendix
Table S6: ZJU-Refer-3D queries.
ID
Query
F01
Select the upper clothing while both upper arms lift outward until they are parallel to the ground.
F02
Select the left hand while the person lowers into a squat.
F03
Select the lower clothing while the left foot moves forward and the right foot remains on the ground.
F04
Select the lower clothing while the left knee bends.
F05
Select the right hand while the right knee bends and the body lowers.
F06
Select the lower clothing while the right knee lifts.
Appendix
Table S7: HuMMan-Refer-4D queries.
ID
Query
Z01
Select the right leg each time the right knee bends.
Z02
Select the left leg while the left knee bends.
Z03
Select the upper clothing while the left upper arm is nearly parallel to the ground.
Z04
Select the right hand while the right upper arm is nearly parallel to the ground.
Z05
Select the right shoe while the whole right leg lifts until it is nearly parallel to the ground.
Z06
Select the left shoe while the left foot is off the floor.
Appendix
Table S8: ZJU-Refer-4D queries.
Role
You are the Appearance Describer. Record visible appearance for the supplied categories from synchronized images.
Task
[…] Describe every supplied category by combining compatible evidence from all supplied views. Preserve visible colors, graphics, readable text, logos, material, shape, texture, coverings, accessories, and feature placement. […]
Rules
Your evidence is the supplied synchronized multiview RGB images. Treat them as one observation set and describe only appearance. […] Combine complementary front, back, and side details […] Left and right refer to the person’s anatomical sides, not the image sides.
Input
{category IDs and names}; {four views from the frame closest to a T-pose}
Appendix
Table S9: Instruction for the appearance describer. […] marks omissions.
Role
You are the 3D Semantic Selector. Map the requested static target to categories supported by Semantic Memory.
Task
Call select_semantic_labels once with the stored categories that match the full static request, or abstain when no supported match exists.
Rules
Use category names, their ordinary meaning and synonyms for structural identity, and stored descriptions for appearance. […] Every requested color, material, texture, logo, graphic, shape, or placement detail needs compatible stored evidence. […] Select only categories present in the supplied inventory.
{"query": "Find the dark shirt with a small front logo and a large rear graphic.", "semantic_inventory": [{"id": 1, "name": "upper_clothing", "text": "A dark shirt has a small front logo and a large graphic across the back.", …}, {"id": 2, "name": "lower_clothing", …}]}
ICL output
{"activation_label_ids": [1], "abstain": false, "reason": "The stored upper-clothing description matches both requested graphics."}
Appendix
Table S10: Instruction and ICL example for the 3D selector. […] marks omissions.
Role
[…] First decide whether the exact query is expressible using the supplied Semantic Memory and body measurements, then return its program or the specific missing support. […]
Task
Return exactly one tool call. Use submit_component_bindings for a fully supported query: supply its answer target, target label IDs, motion or state components, temporal composition, and Registry bindings. Use report_query_unsupported when the supplied inventory or cards cannot express the full request, and identify the missing support.
Rules
The answer target and the subject of a WHEN condition are independent: the condition may describe any requested body part, including a part different from the selected target. […] One component describes one movement or one state of a measurable part or bilateral pair. […] A destination pose qualifies the complete movement that ends in it with state_at_end . […] Report unsupported if the target lacks inventory support, a required anatomical side is missing, no body motion or state is requested, or any required measurement or temporal relation cannot be expressed.
Input
{query}; {inventory}; {template and relation text cards as in Tab. S3 }
ICL input
{"query": "Select the buckle while the body turns counterclockwise.", "semantic_inventory": [{"id": 29, "name": "buckle"}]}
ICL output
{"target_phrase": "the buckle", "components": [{"component_id": "c01", "kind": "motion", "phrase": "the body turns counterclockwise", …}], "root": "c01", "target_label_ids": [29], "bindings_by_component": [{…, "bindings": [{"atom_id": "root_turn_counterclockwise", "part": "body", …}]}], …}
Appendix
Table S11: Instruction and ICL example for the 4D compiler. […] marks omissions.
High-fidelity 3D head avatars have reached near-photorealistic quality. While recent methods enable text-driven manipulation, they struggle to provide fine-grained localized control, often entangling features or lacking geometric consistency. Modifying geometry through natural language currently requires slow per-prompt optimization or compromises identity and rigging. We present ManifoldSplat, the first end-toend framework for language-guided semantic shape editing of animatable 3D Gaussian Splatting avatars reconstructed from monocular videos. By performing edits within the structured FLAME manifold rather than directly optimizing an unstructured Gaussian cloud, we strictly preserve identity and animation. We introduce DeltaRegion, a per-region disentangled Conditional Variational Autoencoder (CVAE) delivering feedforward shape deltas, alongside a refining stage to recover view-consistent details. ManifoldSplat reconstructs and edits an avatar in ~90 seconds on a consumer GPU, rendering at ~800 FPS. Extensive evaluations demonstrate our approach sets a new state-of-the-art in localized prompt alignment, geometric coherence, and identity preservation. Project page and code: https://a-canela.github.io/manifoldsplat/
Antonio Canela, Jordi Sànchez-Riera
Universitat Politècnica de Catalunya · Institut de Robòtica i Informàtica Industrial, CSIC-UPC
High-quality 4D head avatars from one or a few source portraits are central to telepresence, AR/VR, and digital-human interaction. 3D Gaussian Splatting (3DGS) has emerged as the dominant representation, with two complementary regimes (generalizable feed-forward predictors and per-subject refiners) maturing in parallel. However, existing feed-forward predictors are trained on a single dataset family with a hard-coded source count, inheriting the corresponding domain bias. Per-subject refiners require 300K--600K iterations and rely on adaptive densification that destroys upstream Gaussian layouts, preventing the two regimes from sharing a representation end-to-end. To bridge both regimes we propose SpatialAvatar-0 on a shared FLAME-mesh-bound Gaussian representation: a feed-forward generator with a parameter-free K-source mean-pool and a monocular-temporal to multi-view-spatial two-phase schedule that anchors against identity-prior collapse onto the smaller multi-view set. We further introduce a 10K-iter layout-preserving per-subject refinement loop that freezes the FLAME-binding and Gaussian count and replaces densification with a three-component anti-spike regularization. On VFHQ/HDTF cross-domain zero-shot we surpass the in-domain leader GAGAvatar by +1.5 dB PSNR despite never training on either test domain, and on the SplattingAvatar monocular benchmark we lead every reported metric, surpassing the 300K-iter GeoAvatar by +1.3 dB PSNR at up to 60x shorter per-subject schedule than common SOTA baselines. Website: https://spatialwalk.github.io/SpatialAvatar-0.
We present ChromaGS, a method for real-time, language-guided color editing of animatable 3D Gaussian head avatars. Given a trained animatable avatar, users can instantly modify the color of semantic regions through natural language, with edits applied at render time and no retraining required. Our key insight is to augment each Gaussian primitive with learned soft assignments to semantic regions and decompose colors into region-level base colors and Gaussian-level residuals. This decomposition enables coherent color transfer: modifying a region's base color propagates naturally through all associated Gaussians while preserving fine appearance details encoded in residuals. A two-stage language pipeline translates text instructions into target colors, supporting both absolute specifications and relative adjustments. Unlike generative editing methods that may introduce unintended modifications, our approach provides deterministic, precisely localized semantic control. Experiments demonstrate faithful appearance preservation and intuitive interaction across diverse subjects. Project page and code are available at: https://a-canela.github.io/chromags/
Antonio Canela, Jordi Sànchez-Riera
Universitat Polit`ecnica de Catalunya · Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC