Learning Latent Volumetric Dynamics for Multimodal Goal-Conditioned Robot Manipulation of Hair
Authors: Chengyang Zhao, Uksang Yoo, Arkadeep Narayan Chaudhury, Giljoo Nam, Nathan Dennler, Andreea Bobu, Jonathan Francis, Jeffrey Ichnowski, Jean Oh
Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA. · Epic Games, Inc., Pittsburgh, Pennsylvania, USA. · Meta Codec Avatars Lab, Pittsburgh, Pennsylvania, USA. · Bosch Center for Artificial Intelligence, Pittsburgh, Pennsylvania, USA.
Hair care is an essential daily activity for many, yet it remains inaccessible to individuals with limited mobility and challenging for robot systems due to the fine-grained physical structure and complex dynamics of hair. In this work, we present DYMO-Hair, a model-based robot hair care system for generalizable multimodal goal-conditioned hair styling. We introduce a volumetric dynamics learning method that models combing-induced hair deformations through action-conditioned editing in a pre-trained hierarchical 3D latent space. This latent space captures diverse hair states through volumetric occupancy and local strand orientations, while large-scale pre-training, supported by a physics-based hair-combing simulator, enables generalization to previously unseen hairstyles. Coupled with a Model Predictive Path Integral (MPPI) planner, the learned dynamics model enables closed-loop hair styling toward both visual references and natural language instructions as user input. In closed-loop experiments, DYMO-Hair achieves lower geometric error than baseline systems for visual goals and receives significantly higher ratings than VLM-based planners for language goals in a 58-participant user study. Real-world experiments further demonstrate zero-shot transfer to seven physical wigs varying in color, length, and texture, effectively styling hair toward both visual and language goals without real-world fine-tuning. Together, these results represent a step toward more generalizable, flexible, and accessible robot hair styling in the real world. More details are available on: https://dymohair.github.io/.
Figures & tables
Figure 1: Overview. We introduce DYMO-Hair , a modularized, model-based robot hair care system. We propose the first 3D volumetric hair-combing dynamics model, featuring a novel learning approach. It uses an action-conditioned latent state editing mechanism, coupled with a compact 3D latent space over a wide range of hair states, enabled by a hair-combing simulator, for generalizable dynamics modeling. Building on this model, we develop DYMO-Hair with a MPPI-based planner for closed-loop multimodal goal-conditioned hair styling, supporting both visual references and natural language instructions as user input.
Figure 2: Comparison of Hair State Representations. (a) Individual hair strands distinguished by colors. (b) A point cloud at 2K-point resolution, the maximum used for point cloud-based methods in our experiments. (c) 64×64×128 grids with about 5 mm voxel size, where colors denote local strand orientations. Red dashed box: a zoomed-in region. Brown dashed box: the corresponding local strand segments.
Figure 3: DYMO-Hair’s Dynamics Model Overview. Left : State latent space pre-training. A 3D volumetric hierarchical model with vector quantization enables compact compression while preserving detailed representation capability. Right : Dynamics learning. The pre-trained model is adapted to capture hair dynamics in a ControlNet-style framework, formulating dynamics as action-conditioned editing in the pre-trained state latent space. zero : zero-convolution; copy : weight copying for initialization; ⊕ : element-wise addition; ⊗ : 3D attention-based feature fusion. In this phase, only the motion encoding path is trainable, with all pre-trained components frozen.
Figure 4: Constraints for PBD-based Strand-level, Contact-rich Hair Combing Simulation. Top: Each constraint’s formulation and intended effect. Bottom: Simulation results under progressive constraint addition. Starting from the initial state (far left), a combing motion is applied along the white dashed arrow (from red to blue). The red dashed box marks the contact-rich region, with its simulation results shown on the right. With all three constraints, the hair maintains a realistic shape; the twist constraint, in particular, preserves curvature and prevents gravity-induced oversmoothing.
Figure 5: Intent Grounding Module of the Language Goal-conditioned Styling Branch. To translate a high-level language instruction into a physically actionable goal image, we introduce two VLM-based mediation stages around the image generation model. Mediation stage 1: pre-generation reasoning (left): the VLM takes the current observation, the user instruction, and the robot’s combing-only constraints, and translates the instruction into a concrete sequence of actions. Goal image generation (middle): the image generation model is conditioned on this action plan rather than the raw instruction, and queried multiple times for candidate goal images. Mediation stage 2: post-generation selection (right): the VLM is reused as a discriminator, analyzing each candidate on intent alignment, reachability via combing alone, and preservation of intrinsic hair properties, then selecting the final goal image, which is handed to the downstream MPPI planner.
Figure 6: Hairstyles Used in Experiments. Top: 10 synthetic hairstyles used for dynamics model training. Middle: 7 unseen synthetic hairstyles used for dynamics model and closed-loop system evaluation in simulation. Bottom : 7 physical wigs used for real-world closed-loop system evaluation.
Figure 7: Experiment Setup. Left : simulation setup. Right : real-world setup.
Method
CDpoint↓
Errori↓
CDstrand↓
mean
90th
mean
90th
mean
90th
PC-GNN
0.0814
0.1359
13.34
30.16
0.1052
0.1983
V-UNet
0.0792
0.1345
14.25
31.00
0.1047
0.1966
V-FiLM
0.0807
0.1334
13.60
29.59
0.1065
0.2011
Proposed
0.0775
0.1240
12.03
26.12
0.1005
0.1878
Table 1: Dynamics Model Evaluation on Unseen Hairstyles
Figure 8: Qualitative Evaluation of Dynamics Learning on Unseen Hairstyles. From left to right: the initial state, the ground-truth end state, and predicted end states from different methods. Local strand segments are color-mapped by orientation; due to discontinuities in the mapping, strands with similar orientations may occasionally appear in different colors. As a result, local orientation is best assessed from the strand segment geometry rather than the color map.
Split
Method
CDpoint↓
Errori↓
CDstrand↓
mean
90th
mean
90th
mean
90th
Latent Space Pre-training - Reconstruction
Seen
V-1layer
0.0694
0.1015
8.33
18.11
0.0873
0.1485
Proposed
0.0612
0.0959
6.83
14.79
0.0844
0.1467
Unseen
V-1layer
0.0683
0.0992
9.51
19.61
0.0824
0.1398
Proposed
0.0598
0.0936
7.49
15.89
0.0787
0.1362
Table 2: Ablation on Hierarchical Model Structure in Dynamics Learning
Figure 9: Qualitative Evaluation of the Hierarchical Model Structure Ablation in Dynamics Learning. For both seen and unseen hairstyles, we show the ground-truth state alongside reconstructions from V-1layer and the proposed method.
Figure 10: Quantitative Results for Visual Goal-conditioned Hair Styling in Simulation. Experiments are conducted on 7 unseen hairstyles, each with 3 cases repeated 5 times. Left: Error curves over planning steps; solid lines and shaded regions denote the mean and standard deviation across hairstyles, and faster error reduction with lower final error indicates higher effectiveness. Right: Success rate curves w.r.t. error thresholds used to determine success, reflecting the distribution of final errors; curves closer to the top-left correspond to more low-error outcomes and thus better performance. In both panels, error is defined as relative error: the ratio of the current strand-level distance to the initial strand-level distance, which provides a unified metric across hairstyles with varying absolute error magnitudes.
Figure 11: Qualitative Results for Visual Goal-conditioned Hair Styling in Simulation. We show three hard cases on different unseen hairstyles. For each, columns from left to right show the initial state, the goal state, a snapshot of the intermediate planning steps, and the styled end state (annotated with its final relative error).
Figure 12: Rating Results and Qualitative Examples for Language Goal-conditioned Styling. Left: per-construct Likert scores (1–7), each the average of the construct’s three items, for every method-construct cell in each study. Boxes span Q1–Q3 with the median line; whiskers reach 1.5× the interquartile range; open circles are outliers; black dots are LME estimated marginal means. Brackets mark planned pairwise comparisons that are significant under the Holm-corrected LME ( ∗p<.05 , ∗∗p<.01 , ∗∗∗p<.001 ); all planned comparisons are reported in Tab. 3 . Right: two qualitative cases, each with its own hairstyle and language goal, shown in the left and right groups. Within each case, the initial hairstyle and goal are given at the top; the upper block shows each Study 1 method’s styling trajectory, with a snapshot during planning above the final styled state, and the lower block shows two goal image samples generated by each Study 2 variant.
H
Contrast
Δ
p
Δ
p
Study 1: Closed-loop Styling
Outcome Achievement
Progress Consistency
H1
DYMO vs. VLM Planner
+1.60
< .001
+1.34
< .001
H2
DYMO vs. End-to-End VLM
+1.34
< .001
+1.05
< .001
H3
DYMO vs. PC-GNN Model
+0.54
.078
+0.61
.056
Study 2: Semantic Intent Grounding
Intent Alignment
Physical Feasibility
H4
Reasoned vs. Direct
+0.13
1.000
−0.13
1.000
Table 3: Planned Hypothesis Pairwise Comparisons for Language Goal-conditioned Styling. Each study is evaluated on two constructs, reported in the left and right column groups. Method abbreviations: DYMO for DYMO-Hair, Reasoned for Reasoned Grounding , and Direct for Direct Grounding . Δ is the LME-estimated difference in mean rating between the two methods in Likert points (positive: the first method rated higher), and p is Holm-corrected within each study (bold: p<.05 ).
Figure 13: Real-world Hair Styling across Different Hair Types. Zero-shot styling results of DYMO-Hair on 7 physical wigs spanning varied colors, lengths, and textures, under both visual and language goals, with no real-world fine-tuning. Left: visual goal-conditioned styling. Columns show, from left to right, the initial state, the goal state, a snapshot of the intermediate planning steps, and the styled end state. Right: language goal-conditioned styling. The language goal is given above each example, followed by the initial state, a planning snapshot, and the styled end state. Insets in the top-right corner show the front view of each hair state, and each planning snapshot is annotated with the total number of planning steps.
Figure 14: Real-world Visual Goal-conditioned Styling: DYMO-Hair vs. Rule-based System. For each case, the two methods are shown row by row from a nearly identical initial hair state. Each row shows selected keyframes of the styling trajectory, with insets in the top-right corner giving the front view of the hair after the action in each keyframe. The shared visual goal and the per-step error curve for both methods are shown at the bottom of each case; dashed lines connect each keyframe to its corresponding point on the error curve. Error is the relative strand-level distance to the goal, normalized by the initial distance; step 0 is omitted, as this relative error is 1 by definition at the start.
Figure 15: Real-world Language Goal-conditioned Styling: DYMO-Hair vs. End-to-End VLM. For each case, the language goal is given at the top and each method is shown in one row. Within a row, we show the initial state, selected keyframes from the styling trajectory, and the final result; insets in the top-right corner give the front view of the hair after the action in each keyframe. DYMO-Hair first grounds the instruction into a Grounded Goal image, which enables closed-loop planning against an explicit target, and the rightmost plot reports the resulting relative error over planning steps. The End-to-End VLM has no such target and instead predicts actions directly from the language goal.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Construct
α
r1,2
r1,3
r2,3
n
Study 1: Closed-loop Styling
Outcome achievement
0.84
0.88
0.54
0.54
232
Progress consistency
0.61
0.89
0.13
0.08
232
Study 2: Semantic Intent Grounding
Intent alignment
0.69
0.81
0.29
0.28
116
Physical feasibility
0.64
0.63
0.29
0.23
116
Appendix
Table 4: Internal Consistency of the Rating Constructs. Cronbach’s α is computed over all three items of each construct. Items 1 and 2 are positively worded and item 3 is reverse-worded and reverse-scored (listed in Sec. A.2 ). r values are Pearson correlations between item pairs, and n is the number of rated demonstrations.
Study 1: Closed-loop Styling
Outcome Achi.
Progress Cons.
Method
Mean
SD
Mean
SD
VLM Planner
3.50
1.84
3.86
1.27
End-to-End VLM
3.98
1.66
4.22
1.47
PC-GNN Model
4.73
1.55
4.66
1.36
DYMO-Hair
5.30
1.16
5.23
0.92
Appendix
Table 5: Likert Rating Distributions. Mean and standard deviation (SD) of the averaged three-item Likert score (1–7) for each method-construct cell, pooled across tasks, random seeds, and participants.
Construct
Random effects
n
Study 1: Closed-loop Styling
Outcome achievement
(1∣P)+(1∣D)
232
Progress consistency
(1∣P)+(1∣D)
232
Study 2: Semantic Intent Grounding
Intent alignment
(1∣P)+(1∣T)+(1∣D)
116
Physical feasibility
(1∣P)
116
Appendix
Table 6: LME Model-fitting Specification. For each construct we report the random-effect structure selected by the fallback ladder ( P : participant, T : task, D : demonstration) and the number of ratings n it was fit to. The fixed effect method is omitted since it is included in every model. All selected models converged without singularity.
LME
Non-parametric Wilcoxon
H
Pairwise Comparison
Δ
praw
pHolm
r
praw
pHolm
n
Study 1: Closed-loop Styling — Outcome Achievement
H1
DYMO-Hair vs. VLM Planner
+1.60
< .001
< .001
+0.88
.010
.039
16
H2
DYMO-Hair vs. End-to-End VLM
+1.34
< .001
< .001
+0.97
.003
.017
18
H3
DYMO-Hair vs. PC-GNN Model
+0.54
.078
.078
+0.50
.088
.099
16
Study 1: Closed-loop Styling — Progress Consistency
Appendix
Table 7: Full Planned Pairwise Comparison Results for Language Goal-conditioned Styling. Δ is the LME effect-size estimate in Likert points (positive: the first method rated higher). r is the matched-pairs rank-biserial correlation from the participant-paired Wilcoxon signed-rank test, computed over the n participants who rated both methods of the comparison. For each test we report the uncorrected p -value ( praw ) and the Holm-corrected value ( pHolm ), corrected across the six planned comparisons within each study. Significant values ( pHolm<.05 ) are in bold.
Parameter
Value
Parameter
Value
Time step
0.01
Substeps
5
Gravity
[0, 0, -5.0]
Damping
0.99
Particle size
0.01
Linear Density
1.0
Stretch solver iterations
20
Bending solver iterations
20
Twist solver iterations
20
Stretch compliance
0.0
Stretch relaxation
1.0
Bending compliance
1×10−5
Appendix
Table 8: Key Simulation Parameters for Hair Combing