cs.CVOct 17, 2026
SaveHumanSkill-4D: Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time
Organizations: UNIST · NAVER AI Lab
Abstract
We introduce avatar referring segmentation, which selects regions of animatable Gaussian avatars from language queries. A query can also specify a pose or motion to select matching intervals. This lets users interact with avatars for local editing and motion inspection. However, making these selections with existing language Gaussian splatting methods poses two challenges: adjacent parts and garments are hard to separate and may be referred to indirectly (region ambiguity), while fine-grained motions can look alike in RGB (motion ambiguity). We address these challenges with HumanSkill-4D, a training-free framework that shares a multimodal large language model (MLLM) and geometric operators across avatars. For each avatar, a semantic memory links body-part and garment categories to persistent Gaussian regions and appearance descriptions. The MLLM uses category names and descriptions to select categories by appearance, relation, or function. To find matching intervals, it composes operators into programs that check query conditions on driving joint trajectories stored in a motion memory. These category choices and operator outputs make selections traceable, while composing or adding operators supports new motion conditions. To evaluate this task, we introduce three human-annotated benchmarks and show that HumanSkill-4D outperforms the baselines in part segmentation and 3D and 4D referring segmentation, with compact memories and only minutes of setup per avatar.
Figures & tables
Figure 1: Avatar referring segmentation with HumanSkill-4D . (a) A 3D query selects the described region of a Gaussian avatar ( Where ). (b) A 4D query also selects the intervals in which the specified pose or motion occurs ( When ). See our project page for video demos.
Figure 2: HumanSkill-4D overview. (Offline) We build a semantic memory for each avatar, a motion memory for its driving pose sequence, and an operator registry shared across avatars. (Online) The frozen MLLM selects target categories from their names and descriptions, and their stored scores determine the spatial mask. For a 4D query, it also reads the registry’s text cards and composes operators into a motion program, and running the program on the joint trajectories gives the temporal mask. The two masks give the spatiotemporal selection. In this example, the program identifies intervals when both upper arms move outward, and StateAtEnd retains each movement interval whose last frame has both arms horizontal (orange band).
| Method | Venue | mIoU per avatar | Overall | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 313 | 315 | 377 | 386 | 387 | 390 | 392 | 393 | 394 | OA | mAcc | mIoU | ||
| 4D-DRESS | CVPR’24 | 35.9 | 33.9 | 34.7 | 24.7 | 27.5 | 20.0 | 37.5 | 24.9 | 24.9 | 78.6 | 35.7 | 31.2 |
| OpenHuman3D | 3DV’25 | 47.7 | 44.8 | 32.3 | 35.6 | 30.9 | 25.9 | 44.3 | 45.9 | 26.9 | 66.6 | 51.8 | 41.2 |
| OpenHuman4D | BMVC’25 | 48.0 | 37.8 | 40.3 | 31.8 | 27.9 | 26.0 | 43.6 | 41.9 | 29.6 | 77.6 | 52.8 | 40.0 |
| Find3D | ICCV’25 | 17.0 | 13.3 | 13.1 | 13.4 | 8.8 | 8.1 | 13.5 | 12.1 | 10.7 | 14.9 | 31.5 | 12.7 |
| HumanSkill-4D | 73.4 | 77.6 | 74.5 | 65.6 | 52.1 | 50.4 | 65.9 | 67.8 | 48.0 | 94.2 | 79.5 | 75.0 | |
Table 1: Part segmentation on ZJU-Refer-3D. OA, mAcc, and mIoU (%) over 16 categories.
| Method | Part | App. | Rel. | All |
|---|---|---|---|---|
| LangSplat | 57.0 | 68.0 | 4.1 | 56.2 |
| LangSplat † | 71.3 | 72.2 | 4.8 | 61.9 |
| ReferSplat | 28.6 | 47.6 | 5.1 | 37.4 |
| ReferSplat † | 52.1 | 35.4 | 12.7 | 35.3 |
| OpenHuman3D | 40.3 | 54.9 | 3.1 | 44.2 |
| Dr. Splat | 8.5 | 16.2 | 9.1 | 13.6 |
Table 2: 3D referring segmentation on ZJU-Refer-3D. mIoU (%) on part, appearance (App.), and relation/function (Rel.) queries; † uses our parser (i.e., Sapiens2).
Figure 3: Qualitative 3D referring segmentation on ZJU-Refer-3D. Two held-out views and a detail crop per query; selected regions are magenta .
| MLLM | mIoU | vIoU |
|---|---|---|
| Qwen3.6-27B | 82.4 | 44.3 |
| Gemma 4-31B | 55.3 | 45.0 |
| Mistral 3.2-24B | 58.6 | 38.8 |
| Qwen3.8-27B | 82.4 | 45.0 |
Table 4: Ablations on MLLM choice ( left ), semantic category selector ( middle ), and score threshold ( right ). We report 3D mIoU and 4D vIoU (%).
Figure 4: Qualitative 4D referring segmentation on HuMMan-Refer-4D. GT and predicted regions are magenta and intervals green .
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Measurement type | Definition | State templates (20) | Change templates (26) | Coordinate frame |
|---|---|---|---|---|
| Joint angle | Angle at a 3-joint hinge (e.g., shoulder–elbow–wrist) | (7) straight, broad bent, and 5 mutually exclusive bent degrees (slight, partial, right angle, deep, full) | (2) bend; straighten | Root-local |
| Relative position | Point relative to a reference (state) or segment endpoint relative to its start (change), along a fixed-frame axis | (6) above/below, front/behind, left/right | (8) move up/down, forward/backward, left/right, inward/outward | Fixed |
| Displacement | Point or body-root travel along a fixed-frame axis | (0) none | (8) up/down, forward/backward, left/right, inward/outward | Fixed |
| Root rotation | Whole-body pitch and yaw | (1) neutral pitch | (4) lean forward/backward; turn clockwise/counterclockwise | Fixed |
| Segment tilt | Angle of a 2-joint segment relative to gravity | (2) vertical; horizontal | (2) toward vertical; toward horizontal | Gravity |
| Distance | Euclidean distance between 2 body points | (2) near; far | (2) approach; separate | Root-local |
Table S1: Operator registry . Templates grouped by measurement; counts in parentheses.
Figure S1: Running-query templates. Move outward (left) and horizontal (right) on both upper arms.
Figure S2: Operator measurements of the running query. The top panel shows how far each elbow lies outward from its shoulder ( ), and the bottom panel shows the angle of each upper arm from vertical ( ). Solid lines are the left arm and dashed lines the right, and amber marks the frames each operator returns.
| Relation | Inputs | Output |
|---|---|---|
| CoOccurrence | 2 or more movements | occurrences formed by intervals shared by every movement |
| AllStates | 2 or more states | frames shared by all states |
| Sequence | movements in order; optional leading state | span from the first to the last movement, linking each last occurrence to its nearest eligible predecessors with no maximum gap; consecutive movements may share a turning frame; a leading state must end before the first movement starts and does not extend the span |
| Alternation | 2 movements A and B | spans of alternating chains of 3 or more occurrences; consecutive movements have disjoint interiors and may share a turning frame |
| StateAtEnd | a movement and a state | occurrences whose last frame lies in the state |
| StateThroughout | a movement and a state | occurrences fully covered by the state |
Table S2: Temporal relations.
Figure S3: Execution record of the running query. Bars mark the frames returned by each operator and relation; dashed lines and bodies mark the last frame of each co-occurrence.
| Card (kind) | Card summary |
|---|---|
| move outward (change) | Measures: the segment endpoint, relative to its start, moves toward its anatomical outer side. Parts: arm, forearm, leg, lower leg, thigh, upper arm. |
| horizontal (state) | Measures: the segment is horizontal relative to gravity. |
| CoOccurrence (relation) | Inputs: two or more movements. Meaning: the movements must hold on the same frames. Returns: every shared portion on which all of the movements hold. |
| AllStates (relation) | Inputs: two or more states. Meaning: the states must hold on the same frames. Returns: the frames on which every state holds. |
| StateAtEnd (relation) | Inputs: a movement, then a state. Meaning: the state describes the pose in which the movement ends. Returns: the movement occurrence whose last frame lies inside the state. |
Table S3: Text cards used for the running query.
| Method | Segmentation | Interpretation | Stored information |
|---|---|---|---|
| 3D referring segmentation | |||
| LangSplat | SAM | CLIP | Features + networks |
| ReferSplat | Grounded-SAM | BERT + attention | Features + networks |
| OpenHuman3D | SAM | HumanCLIP | Masks + features |
| HumanSkill-4D | Sapiens2 | MLLM | Scores + text |
| 4D referring segmentation | |||
Table S4: Method components for the computational cost comparison in Tab. 5 . Stored information lists component types. Segmentation: 2D mask source; : none.
| Model | mIoU | vIoU |
|---|---|---|
| Gemma 4 E2B | 60.2 | 0.0 |
| Ministral 3 8B | 63.1 | 20.4 |
| Qwen3-VL-8B | 78.4 | 16.2 |
| Phi-4 | 49.9 | 2.6 |
Table S5: Smaller MLLMs. 3D mIoU and 4D vIoU (%).
Figure S4: Qualitative part segmentation on ZJU-Refer-3D.
Figure S5: Qualitative 3D referring segmentation on ZJU-Refer-3D. † uses Sapiens2 masks.
Figure S6: Qualitative 3D referring segmentation on ZJU-Refer-3D. † uses Sapiens2 masks.
Figure S7: Qualitative 4D referring segmentation on HuMMan-Refer-4D.
Figure S8: Qualitative 4D referring segmentation on ZJU-Refer-4D.
Figure S9: Qualitative 4D referring segmentation on ZJU-Refer-4D.
Figure S10: Qualitative 4D referring segmentation on ZJU-Refer-4D.
| ID | Query |
|---|---|
| Q01 | Select both hands. |
| Q02 | Select both arms. |
| Q03 | Select both shoes. |
| Q04 | Select the upper clothing and lower clothing. |
| Q05 | Select the garment with a zipper down the front. |
| Q06 | Select the yellow garment. |
Table S6: ZJU-Refer-3D queries.
| ID | Query |
|---|---|
| F01 | Select the upper clothing while both upper arms lift outward until they are parallel to the ground. |
| F02 | Select the left hand while the person lowers into a squat. |
| F03 | Select the lower clothing while the left foot moves forward and the right foot remains on the ground. |
| F04 | Select the lower clothing while the left knee bends. |
| F05 | Select the right hand while the right knee bends and the body lowers. |
| F06 | Select the lower clothing while the right knee lifts. |
Table S7: HuMMan-Refer-4D queries.
| ID | Query |
|---|---|
| Z01 | Select the right leg each time the right knee bends. |
| Z02 | Select the left leg while the left knee bends. |
| Z03 | Select the upper clothing while the left upper arm is nearly parallel to the ground. |
| Z04 | Select the right hand while the right upper arm is nearly parallel to the ground. |
| Z05 | Select the right shoe while the whole right leg lifts until it is nearly parallel to the ground. |
| Z06 | Select the left shoe while the left foot is off the floor. |
Table S8: ZJU-Refer-4D queries.
| Role | You are the Appearance Describer. Record visible appearance for the supplied categories from synchronized images. |
|---|---|
| Task | […] Describe every supplied category by combining compatible evidence from all supplied views. Preserve visible colors, graphics, readable text, logos, material, shape, texture, coverings, accessories, and feature placement. […] |
| Rules | Your evidence is the supplied synchronized multiview RGB images. Treat them as one observation set and describe only appearance. […] Combine complementary front, back, and side details […] Left and right refer to the person’s anatomical sides, not the image sides. |
| Input | {category IDs and names}; {four views from the frame closest to a T-pose} |
Table S9: Instruction for the appearance describer. […] marks omissions.
| Role | You are the 3D Semantic Selector. Map the requested static target to categories supported by Semantic Memory. |
|---|---|
| Task | Call select_semantic_labels once with the stored categories that match the full static request, or abstain when no supported match exists. |
| Rules | Use category names, their ordinary meaning and synonyms for structural identity, and stored descriptions for appearance. […] Every requested color, material, texture, logo, graphic, shape, or placement detail needs compatible stored evidence. […] Select only categories present in the supplied inventory. |
| Input | {query}; {inventory rows: category ID, name, description, status} |
| ICL input | {"query": "Find the dark shirt with a small front logo and a large rear graphic.", "semantic_inventory": [{"id": 1, "name": "upper_clothing", "text": "A dark shirt has a small front logo and a large graphic across the back.", …}, {"id": 2, "name": "lower_clothing", …}]} |
| ICL output | {"activation_label_ids": [1], "abstain": false, "reason": "The stored upper-clothing description matches both requested graphics."} |
Table S10: Instruction and ICL example for the 3D selector. […] marks omissions.
| Role | […] First decide whether the exact query is expressible using the supplied Semantic Memory and body measurements, then return its program or the specific missing support. […] |
|---|---|
| Task | Return exactly one tool call. Use submit_component_bindings for a fully supported query: supply its answer target, target label IDs, motion or state components, temporal composition, and Registry bindings. Use report_query_unsupported when the supplied inventory or cards cannot express the full request, and identify the missing support. |
| Rules | The answer target and the subject of a WHEN condition are independent: the condition may describe any requested body part, including a part different from the selected target. […] One component describes one movement or one state of a measurable part or bilateral pair. […] A destination pose qualifies the complete movement that ends in it with state_at_end . […] Report unsupported if the target lacks inventory support, a required anatomical side is missing, no body motion or state is requested, or any required measurement or temporal relation cannot be expressed. |
| Input | {query}; {inventory}; {template and relation text cards as in Tab. S3 } |
| ICL input | {"query": "Select the buckle while the body turns counterclockwise.", "semantic_inventory": [{"id": 29, "name": "buckle"}]} |
| ICL output | {"target_phrase": "the buckle", "components": [{"component_id": "c01", "kind": "motion", "phrase": "the body turns counterclockwise", …}], "root": "c01", "target_label_ids": [29], "bindings_by_component": [{…, "bindings": [{"atom_id": "root_turn_counterclockwise", "part": "body", …}]}], …} |
Table S11: Instruction and ICL example for the 4D compiler. […] marks omissions.