Learning Latent Volumetric Dynamics for Multimodal Goal-Conditioned Robot Manipulation of Hair
Authors: Chengyang Zhao, Uksang Yoo, Arkadeep Narayan Chaudhury, Giljoo Nam, Nathan Dennler, Andreea Bobu, Jonathan Francis, Jeffrey Ichnowski, Jean Oh
Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA. · Epic Games, Inc., Pittsburgh, Pennsylvania, USA. · Meta Codec Avatars Lab, Pittsburgh, Pennsylvania, USA. · Bosch Center for Artificial Intelligence, Pittsburgh, Pennsylvania, USA.
Hair care is an essential daily activity for many, yet it remains inaccessible to individuals with limited mobility and challenging for robot systems due to the fine-grained physical structure and complex dynamics of hair. In this work, we present DYMO-Hair, a model-based robot hair care system for generalizable multimodal goal-conditioned hair styling. We introduce a volumetric dynamics learning method that models combing-induced hair deformations through action-conditioned editing in a pre-trained hierarchical 3D latent space. This latent space captures diverse hair states through volumetric occupancy and local strand orientations, while large-scale pre-training, supported by a physics-based hair-combing simulator, enables generalization to previously unseen hairstyles. Coupled with a Model Predictive Path Integral (MPPI) planner, the learned dynamics model enables closed-loop hair styling toward both visual references and natural language instructions as user input. In closed-loop experiments, DYMO-Hair achieves lower geometric error than baseline systems for visual goals and receives significantly higher ratings than VLM-based planners for language goals in a 58-participant user study. Real-world experiments further demonstrate zero-shot transfer to seven physical wigs varying in color, length, and texture, effectively styling hair toward both visual and language goals without real-world fine-tuning. Together, these results represent a step toward more generalizable, flexible, and accessible robot hair styling in the real world. More details are available on: https://dymohair.github.io/.
Figures & tables
Figure 1: Overview. We introduce DYMO-Hair , a modularized, model-based robot hair care system. We propose the first 3D volumetric hair-combing dynamics model, featuring a novel learning approach. It uses an action-conditioned latent state editing mechanism, coupled with a compact 3D latent space over a wide range of hair states, enabled by a hair-combing simulator, for generalizable dynamics modeling. Building on this model, we develop DYMO-Hair with a MPPI-based planner for closed-loop multimodal goal-conditioned hair styling, supporting both visual references and natural language instructions as user input.
Figure 2: Comparison of Hair State Representations. (a) Individual hair strands distinguished by colors. (b) A point cloud at 2K-point resolution, the maximum used for point cloud-based methods in our experiments. (c) 64×64×128 grids with about 5 mm voxel size, where colors denote local strand orientations. Red dashed box: a zoomed-in region. Brown dashed box: the corresponding local strand segments.
Figure 3: DYMO-Hair’s Dynamics Model Overview. Left : State latent space pre-training. A 3D volumetric hierarchical model with vector quantization enables compact compression while preserving detailed representation capability. Right : Dynamics learning. The pre-trained model is adapted to capture hair dynamics in a ControlNet-style framework, formulating dynamics as action-conditioned editing in the pre-trained state latent space. zero : zero-convolution; copy : weight copying for initialization; ⊕ : element-wise addition; ⊗ : 3D attention-based feature fusion. In this phase, only the motion encoding path is trainable, with all pre-trained components frozen.
Figure 4: Constraints for PBD-based Strand-level, Contact-rich Hair Combing Simulation. Top: Each constraint’s formulation and intended effect. Bottom: Simulation results under progressive constraint addition. Starting from the initial state (far left), a combing motion is applied along the white dashed arrow (from red to blue). The red dashed box marks the contact-rich region, with its simulation results shown on the right. With all three constraints, the hair maintains a realistic shape; the twist constraint, in particular, preserves curvature and prevents gravity-induced oversmoothing.
Figure 5: Intent Grounding Module of the Language Goal-conditioned Styling Branch. To translate a high-level language instruction into a physically actionable goal image, we introduce two VLM-based mediation stages around the image generation model. Mediation stage 1: pre-generation reasoning (left): the VLM takes the current observation, the user instruction, and the robot’s combing-only constraints, and translates the instruction into a concrete sequence of actions. Goal image generation (middle): the image generation model is conditioned on this action plan rather than the raw instruction, and queried multiple times for candidate goal images. Mediation stage 2: post-generation selection (right): the VLM is reused as a discriminator, analyzing each candidate on intent alignment, reachability via combing alone, and preservation of intrinsic hair properties, then selecting the final goal image, which is handed to the downstream MPPI planner.
Figure 6: Hairstyles Used in Experiments. Top: 10 synthetic hairstyles used for dynamics model training. Middle: 7 unseen synthetic hairstyles used for dynamics model and closed-loop system evaluation in simulation. Bottom : 7 physical wigs used for real-world closed-loop system evaluation.
Figure 7: Experiment Setup. Left : simulation setup. Right : real-world setup.
Method
CDpoint↓
Errori↓
CDstrand↓
mean
90th
mean
90th
mean
90th
PC-GNN
0.0814
0.1359
13.34
30.16
0.1052
0.1983
V-UNet
0.0792
0.1345
14.25
31.00
0.1047
0.1966
V-FiLM
0.0807
0.1334
13.60
29.59
0.1065
0.2011
Proposed
0.0775
0.1240
12.03
26.12
0.1005
0.1878
Table 1: Dynamics Model Evaluation on Unseen Hairstyles
Figure 8: Qualitative Evaluation of Dynamics Learning on Unseen Hairstyles. From left to right: the initial state, the ground-truth end state, and predicted end states from different methods. Local strand segments are color-mapped by orientation; due to discontinuities in the mapping, strands with similar orientations may occasionally appear in different colors. As a result, local orientation is best assessed from the strand segment geometry rather than the color map.
Split
Method
CDpoint↓
Errori↓
CDstrand↓
mean
90th
mean
90th
mean
90th
Latent Space Pre-training - Reconstruction
Seen
V-1layer
0.0694
0.1015
8.33
18.11
0.0873
0.1485
Proposed
0.0612
0.0959
6.83
14.79
0.0844
0.1467
Unseen
V-1layer
0.0683
0.0992
9.51
19.61
0.0824
0.1398
Proposed
0.0598
0.0936
7.49
15.89
0.0787
0.1362
Table 2: Ablation on Hierarchical Model Structure in Dynamics Learning
Figure 9: Qualitative Evaluation of the Hierarchical Model Structure Ablation in Dynamics Learning. For both seen and unseen hairstyles, we show the ground-truth state alongside reconstructions from V-1layer and the proposed method.
Figure 10: Quantitative Results for Visual Goal-conditioned Hair Styling in Simulation. Experiments are conducted on 7 unseen hairstyles, each with 3 cases repeated 5 times. Left: Error curves over planning steps; solid lines and shaded regions denote the mean and standard deviation across hairstyles, and faster error reduction with lower final error indicates higher effectiveness. Right: Success rate curves w.r.t. error thresholds used to determine success, reflecting the distribution of final errors; curves closer to the top-left correspond to more low-error outcomes and thus better performance. In both panels, error is defined as relative error: the ratio of the current strand-level distance to the initial strand-level distance, which provides a unified metric across hairstyles with varying absolute error magnitudes.
Figure 11: Qualitative Results for Visual Goal-conditioned Hair Styling in Simulation. We show three hard cases on different unseen hairstyles. For each, columns from left to right show the initial state, the goal state, a snapshot of the intermediate planning steps, and the styled end state (annotated with its final relative error).
Figure 12: Rating Results and Qualitative Examples for Language Goal-conditioned Styling. Left: per-construct Likert scores (1–7), each the average of the construct’s three items, for every method-construct cell in each study. Boxes span Q1–Q3 with the median line; whiskers reach 1.5× the interquartile range; open circles are outliers; black dots are LME estimated marginal means. Brackets mark planned pairwise comparisons that are significant under the Holm-corrected LME ( ∗p<.05 , ∗∗p<.01 , ∗∗∗p<.001 ); all planned comparisons are reported in Tab. 3 . Right: two qualitative cases, each with its own hairstyle and language goal, shown in the left and right groups. Within each case, the initial hairstyle and goal are given at the top; the upper block shows each Study 1 method’s styling trajectory, with a snapshot during planning above the final styled state, and the lower block shows two goal image samples generated by each Study 2 variant.
H
Contrast
Δ
p
Δ
p
Study 1: Closed-loop Styling
Outcome Achievement
Progress Consistency
H1
DYMO vs. VLM Planner
+1.60
< .001
+1.34
< .001
H2
DYMO vs. End-to-End VLM
+1.34
< .001
+1.05
< .001
H3
DYMO vs. PC-GNN Model
+0.54
.078
+0.61
.056
Study 2: Semantic Intent Grounding
Intent Alignment
Physical Feasibility
H4
Reasoned vs. Direct
+0.13
1.000
−0.13
1.000
Table 3: Planned Hypothesis Pairwise Comparisons for Language Goal-conditioned Styling. Each study is evaluated on two constructs, reported in the left and right column groups. Method abbreviations: DYMO for DYMO-Hair, Reasoned for Reasoned Grounding , and Direct for Direct Grounding . Δ is the LME-estimated difference in mean rating between the two methods in Likert points (positive: the first method rated higher), and p is Holm-corrected within each study (bold: p<.05 ).
Figure 13: Real-world Hair Styling across Different Hair Types. Zero-shot styling results of DYMO-Hair on 7 physical wigs spanning varied colors, lengths, and textures, under both visual and language goals, with no real-world fine-tuning. Left: visual goal-conditioned styling. Columns show, from left to right, the initial state, the goal state, a snapshot of the intermediate planning steps, and the styled end state. Right: language goal-conditioned styling. The language goal is given above each example, followed by the initial state, a planning snapshot, and the styled end state. Insets in the top-right corner show the front view of each hair state, and each planning snapshot is annotated with the total number of planning steps.
Figure 14: Real-world Visual Goal-conditioned Styling: DYMO-Hair vs. Rule-based System. For each case, the two methods are shown row by row from a nearly identical initial hair state. Each row shows selected keyframes of the styling trajectory, with insets in the top-right corner giving the front view of the hair after the action in each keyframe. The shared visual goal and the per-step error curve for both methods are shown at the bottom of each case; dashed lines connect each keyframe to its corresponding point on the error curve. Error is the relative strand-level distance to the goal, normalized by the initial distance; step 0 is omitted, as this relative error is 1 by definition at the start.
Figure 15: Real-world Language Goal-conditioned Styling: DYMO-Hair vs. End-to-End VLM. For each case, the language goal is given at the top and each method is shown in one row. Within a row, we show the initial state, selected keyframes from the styling trajectory, and the final result; insets in the top-right corner give the front view of the hair after the action in each keyframe. DYMO-Hair first grounds the instruction into a Grounded Goal image, which enables closed-loop planning against an explicit target, and the rightmost plot reports the resulting relative error over planning steps. The End-to-End VLM has no such target and instead predicts actions directly from the language goal.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Construct
α
r1,2
r1,3
r2,3
n
Study 1: Closed-loop Styling
Outcome achievement
0.84
0.88
0.54
0.54
232
Progress consistency
0.61
0.89
0.13
0.08
232
Study 2: Semantic Intent Grounding
Intent alignment
0.69
0.81
0.29
0.28
116
Physical feasibility
0.64
0.63
0.29
0.23
116
Appendix
Table 4: Internal Consistency of the Rating Constructs. Cronbach’s α is computed over all three items of each construct. Items 1 and 2 are positively worded and item 3 is reverse-worded and reverse-scored (listed in Sec. A.2 ). r values are Pearson correlations between item pairs, and n is the number of rated demonstrations.
Study 1: Closed-loop Styling
Outcome Achi.
Progress Cons.
Method
Mean
SD
Mean
SD
VLM Planner
3.50
1.84
3.86
1.27
End-to-End VLM
3.98
1.66
4.22
1.47
PC-GNN Model
4.73
1.55
4.66
1.36
DYMO-Hair
5.30
1.16
5.23
0.92
Appendix
Table 5: Likert Rating Distributions. Mean and standard deviation (SD) of the averaged three-item Likert score (1–7) for each method-construct cell, pooled across tasks, random seeds, and participants.
Construct
Random effects
n
Study 1: Closed-loop Styling
Outcome achievement
(1∣P)+(1∣D)
232
Progress consistency
(1∣P)+(1∣D)
232
Study 2: Semantic Intent Grounding
Intent alignment
(1∣P)+(1∣T)+(1∣D)
116
Physical feasibility
(1∣P)
116
Appendix
Table 6: LME Model-fitting Specification. For each construct we report the random-effect structure selected by the fallback ladder ( P : participant, T : task, D : demonstration) and the number of ratings n it was fit to. The fixed effect method is omitted since it is included in every model. All selected models converged without singularity.
LME
Non-parametric Wilcoxon
H
Pairwise Comparison
Δ
praw
pHolm
r
praw
pHolm
n
Study 1: Closed-loop Styling — Outcome Achievement
H1
DYMO-Hair vs. VLM Planner
+1.60
< .001
< .001
+0.88
.010
.039
16
H2
DYMO-Hair vs. End-to-End VLM
+1.34
< .001
< .001
+0.97
.003
.017
18
H3
DYMO-Hair vs. PC-GNN Model
+0.54
.078
.078
+0.50
.088
.099
16
Study 1: Closed-loop Styling — Progress Consistency
Appendix
Table 7: Full Planned Pairwise Comparison Results for Language Goal-conditioned Styling. Δ is the LME effect-size estimate in Likert points (positive: the first method rated higher). r is the matched-pairs rank-biserial correlation from the participant-paired Wilcoxon signed-rank test, computed over the n participants who rated both methods of the comparison. For each test we report the uncorrected p -value ( praw ) and the Holm-corrected value ( pHolm ), corrected across the six planned comparisons within each study. Significant values ( pHolm<.05 ) are in bold.
Parameter
Value
Parameter
Value
Time step
0.01
Substeps
5
Gravity
[0, 0, -5.0]
Damping
0.99
Particle size
0.01
Linear Density
1.0
Stretch solver iterations
20
Bending solver iterations
20
Twist solver iterations
20
Stretch compliance
0.0
Stretch relaxation
1.0
Bending compliance
1×10−5
Appendix
Table 8: Key Simulation Parameters for Hair Combing
Hair stroking is common in daily grooming and personal care, and is also widely used in hair-product evaluation, motivating robots with similar physical interaction capabilities. Existing robotic hair-care and surface-following methods mainly rely on trajectory planning, compliance, force regulation, or tactile-conditioned policies, but deformable hair can remain in contact while gradually drifting across the end-effector, making local interaction difficult to regulate. We propose TacHair, a tactile contact-distribution guided online correction framework that represents high-resolution tactile observations as a spatial hair-contact distribution. A visuotactile imitation policy generates the nominal stroking motion, while a separately trained residual module corrects local contact deviations, separating task progression from contact recovery. We evaluate TacHair in 525 real-robot trials across five head geometries and three hair conditions. A successful stroke requires both sufficient task progression and contact maintenance; our method improves success from 42.9% to 62.3% and contact maintenance from 59.4% to 88.0% over the same visuotactile policy without correction. These results demonstrate spatial tactile contact distributions as an effective feedback representation for contact-preserving interaction with deformable and visually occluded surfaces. Demos, code, and datasets are available at https://tachair.github.io.
Ruiyi Hu, Yongqiang Zhao, Daniel Bak +3
King’s College London, London WC2R 2LS, United Kingdom
Hair simulation and rendering are challenging due to complex strand dynamics, diverse material properties, and intricate light-hair interactions. Recent video diffusion models can generate high-quality videos, but they lack fine-grained control over hair dynamics. We present ControlHair, a hybrid framework that integrates a physics simulator with conditional video diffusion to enable precise and controllable dynamic hair rendering. ControlHair adopts a three-stage pipeline: it first encodes physics conditions into per-frame geometry using a simulator, then extracts per-frame control signals, and finally feeds control signals into a video diffusion model to generate videos with desired hair dynamics. This cascaded design decouples physics reasoning from video generation, supports diverse physics, and makes training the video diffusion model easy. Trained on a curated 10K video dataset, ControlHair outperforms text- and pose-conditioned baselines, delivering precisely controlled hair dynamics. We also demonstrate three use cases of ControlHair, including dynamic hairstyle try-on, bullet-time effects, and cinemagraphic. Project page: https://linwk20.github.io/controlhair-web.
Weikai Lin, Haoxiang Li, Yuhao Zhu
University of Rochester, USA · Pixocial Technology, USA
Modeling 3D hair strands is challenging due to complex hairstyle geometry, scarce high-quality training data, and incomplete or noisy input observations. Foundation models offer broader hairstyle priors, but their outputs are not directly usable for strand-based rendering and simulation. Existing approaches require costly per-instance optimization to fit hair priors to meshes from foundation models and 2D orientations from specialized predictors trained on limited hair data, restricting efficiency and generalization. We present HairFlow, a multimodal conditioned diffusion framework for efficient and controllable hair strands modeling. HairFlow conditions strand generation on two complementary representations: a coarse mesh that constrains global shape and multi-view sketches that specify local strand flow. HairFlow combines an Orientation VAE with a conditioned diffusion transformer to generate volumetric fields that are traced into explicit strands. To overcome the lack of paired sketch-strand training data, we introduce an automatic synthesis pipeline that generates geometrically aligned multi-view sketches with diverse styles representative of inference-time inputs. Experiments show that HairFlow generates plausible 3D strands that faithfully follow the input conditions across diverse hairstyles, substantially reduces inference time, and supports flexible multimodal control.