We introduce Material Coating, a novel image editing task that simulates applying a thin material layer onto an object while preserving its underlying coarse and fine geometry. Material coating is fundamentally different from existing "material transfer" methods, which are designed to replace an object's intrinsic material, often overwriting fine details. To address this new task, we construct a large-scale synthetic dataset (110K images) of 3D objects with varied, physically-based coatings, named DataCoat110K. We then propose CoatFusion, a novel architecture that enables this task by conditioning a diffusion model on both a 2D albedo texture and granular, PBR-style parametric controls, including roughness, metalness, transmission, and a key thickness parameter. Experiments and user studies show CoatFusion produces realistic, controllable coatings and significantly outperforms existing material editing and transfer methods on this new task.
Figures & tables
Figure 1. Overview. Our method, CoatFusion, coats objects in input images with materials defined by albedo images, masks and parametric controls. Top Left: Objects coated with translucent materials: varnish and colorful glaze. Top Right: Objects with complex geometry covered by drawings and patterns, showing implicit understanding of texture warping. Bottom Left: Varying levels of coating material thickness. Bottom Right: Metallic and dielectric material coatings. Note: text labels are shown for the reader’s convenience only and are not part of the model’s input. Four labelled groups of before-and-after photographs. Top left, translucent coatings: a terracotta hanging planter becomes glossy purple, and a matte wooden chess rook gains a wet-looking varnish; each is accompanied by its flat albedo swatch and a binary object mask. Top right, texture wrapping: a cream ribbed knit sweater held up by a person is covered with a large violet floral pattern that follows the folds of the fabric, and a grey wicker basket is covered with a teal and orange geometric pattern that follows the individual reeds of the weave. Bottom left, thickness: a red weathered fire hydrant coated in yellow at thickness zero, where the pitted and chipped surface still shows through, and at thickness one, where the surface is smooth. Bottom right, an orange vintage car on a Havana street coated as a matte dielectric and as a mirror-like metallic.
Figure 2. Material Transfer vs. Material Coating. The latter task aims to maintain the geometry and fine-grained texture details of the original object, as if a layer of paint was applied on top of the geometry. A two-by-two grid of photographs. Each column pair shows a small input photo and an exemplar swatch on the left and a large output on the right. Top row: a figurine of a large-eared, big-eyed green creature in a brown robe, with a mint-glazed ceramic mug as the exemplar. Material transfer by ZeST reinterprets the figurine as a glazed ceramic vessel, blurring the face and losing the folds of the robe; coating keeps the ears, the eyes, the folds of the robe and the embossed lettering on the base, tinting them mint green. Bottom row: a rusty teal propane cylinder with a plain yellow swatch. Transfer produces a ribbed ceramic pot that no longer resembles the cylinder; coating keeps the valve, the welded seam and the pitted rust while turning the surface yellow.
Figure 3. Implicit geometry-aware texture wrapping . The model learns to map canonical albedo textures onto complex object geometries. Two rows of three photographs, headed by two flat albedo textures: a violet and grey floral pattern and a cartoon black-and-white cat on a beige ground. Top row: a plain cardboard box, with its binary mask inset, then the same box printed with the floral pattern continuing correctly across the top, front and side faces, then the same box with the cat rendered once across the front and wrapping around the corner onto the side. Bottom row: a red rubber ball in a parking lot, with its mask inset, then the same ball with the floral pattern curving over the sphere, then with the cat stretched around its curvature.
Figure 4. Samples from the held-out test set, used as our benchmark dataset. A table of four rendered examples. Columns are the coating parameters, the source image, the binary coating mask, the flat albedo texture, and the resulting coated render. Rows: a woven wicker tray coated with a teal bird illustration; a wooden armchair given a transmissive coating at thickness 0.713 that softens its outline; a slatted media console coated with a blue watercolor wash; and a wooden sideboard given a highly metallic coating carrying a blue swirl logo.
Figure 5. Thickness variation. A sample visualization showing the variation of thickness present in our dataset. A row of inputs above a row of four renders. The inputs are a binary mask, a weathered stone slab, a solid cyan albedo, and the settings metallic one, transmission zero, roughness zero. The four renders show thickness increasing from zero to one along an arrow: at low thickness the cyan metallic coat follows every pit and crack of the stone, and by thickness one the surface is a smooth cyan shell in which the substrate relief is largely flattened out.
Figure 6. Thickness-dependent mappings of rendering parameters. Increasing thickness reduces Normal Strength non-linearly, suppressing substrate microstructure and producing a smoother coated surface; increases the Volume Absorption Factor for transmissive coatings, making them less transparent; and raises the Opaque Albedo Blending weight, moving opaque coatings non-linearly toward the coating albedo while preserving more of the substrate albedo at low thickness values. Formulas are detailed in the Appendix. A line plot with thickness t from 0 to 1 on the horizontal axis and normalized value from 0 to 1 on the vertical axis. A blue solid curve, Normal Strength, stays near 1 until about t equals 0.3 and then falls in an S shape to about 0.3 at t equals 1. A red dashed curve, Opaque Albedo Blending, rises steeply from about 0.2 and saturates near 1 by t equals 0.6. A green solid line, Volume Absorption Factor, rises roughly linearly from about 0.2 to 1.
Figure 7. Varying thickness : Examples of applying a coating with minimal and maximal thickness levels. Note the increase in smoothness for the opaque coating (top rows), and the increasing visibility of the bluish coating color for the transmissive coating (bottom rows). Two labelled blocks, each showing an input beside results at thickness zero and thickness one. In the opaque block, a wooden cutting board floating over water receives a pink graffiti coating whose painted numeral stays crisp while the wood grain disappears at high thickness, and a carved stone knotwork relief coated in pink keeps its deep carving at thickness zero but flattens into smooth pink at thickness one. In the translucent block, a cyan albedo swatch is applied to the same stone knotwork and to the wooden board: at thickness zero the coating is nearly invisible, and at thickness one both objects take on a deep blue-green tint through which the carving and the grain remain visible.
Figure 8. Examples from the dataset generation pipeline, generated in Blender. Each row shows (from left to right) the coating parameters, the original image, the coating mask, the albedo texture, and the resulting image. A table of five rendered examples with columns for the coating parameters, source image, binary coating mask, albedo texture, and result. Rows: a pale stone bench coated with a metallic green leaf logo; a carved stone capital with a transmissive coating at thickness zero, which leaves it nearly unchanged; the same capital at thickness one, now tinted pink; a cardboard bale coated with a pink daisy pattern at roughness one, giving a matte finish; and the same bale at roughness zero, giving a glossier one. The masks are irregular, covering only part of each object.
Figure 9. Training and losses. The model learns to compose and decompose the coating on the substrate. Training steps alternate between compose and decompose tracks. For the compose track (left): we first mask and predict the coating C∗ . Next, we mask the substrate S and albedo A and predict them using C∗ . The decompose track (right) runs in the reverse order. See Section 4.3 for details. Note that the mask channel is omitted to save space (instead, masked areas are filled with gray on the input triptychs). A flow diagram over an example whose substrate S is a carved animal skull, whose albedo A is a dark red and cream damask pattern, and whose coated result C is the skull covered in that damask. The complete triptych at the top splits into two branches. On the left, the compose track: the C panel is masked out in grey, the operator F sub C predicts it, and the prediction C star is outlined in green; the result is then fed back with S and A masked, and F sub D recovers them as S star and A star, outlined in red. On the right, the decompose track runs the same steps in the opposite order, producing S prime and A prime outlined in red and then C prime outlined in green. A reconstruction loss L sub rec links the two first-stage predictions and a cycle loss L sub cyc links the two second-stage predictions.
Figure 10. Thickness comparison . Increasing thickness progressively smooths substrate details and reduces transmissivity. Inputs at the top: a photograph of a weathered blue-grey wooden door set in an ochre plaster wall, a solid magenta albedo, and a binary mask covering the door. Below, six results at thickness 0, 0.2, 0.4, 0.6, 0.8 and 1, each with a circular zoom inset on the door panel. At low thickness the pink coating still reveals the flaking paint, wood grain and nail marks; as thickness increases these fade until the door reads as a smooth, uniformly pink surface with only the plank seams remaining.
Figure 11. Roughness control . The coating appearance changes continuously with the roughness parameter while preserving the underlying geometry. Inputs at the top: a stack of three turned wooden bowls in raking light, a solid orange albedo, and a mask covering only the middle bowl. Below, three results at roughness 0, 0.5 and 1, each with a circular zoom inset. At roughness zero the coated middle bowl carries a broad specular highlight and looks lacquered; at roughness one it is matte and diffuse. The upper and lower bowls and the cast shadows are unchanged throughout.
Figure 12. Comparison of CoatFusion with baselines on the material coating task. From top to bottom: applying thick translucent varnish to wooden cabinet doors, thick matte paint with a dotted pattern on clay mugs, thin metallic coating on the rightmost figure. Our method best adheres to the specified material characteristics, while preserving the fine details of the underlying geometry. A grid with three examples in rows and eight columns: the input, our result, and results from MaterialFusion, ZeST, MatSwap, RGB to X, Photoshop Blend If, and Photoshop Color Blend. The input cell of each row holds the source photograph, the requested material traits as text, the albedo swatch and the mask. Row one applies a thick translucent varnish to the doors of a wooden sideboard; the transfer baselines repaint or restyle the cabinet, while ours darkens and glosses the doors only. Row two applies a thick matte blue dotted paint to a stack of ceramic cups; several baselines scatter the dots over the whole stack or ignore them, while ours confines them to the cups and follows their curvature. Row three applies a thin green metallic coating to the rightmost of a row of wooden pawns; most baselines recolor several pawns or lose the wood grain.
Image PSNR ↑
Method
All
Uniform Albedos
Pattern Albedos
Marble
19.70
19.89
19.57
ZeST
20.51
21.33
19.93
MatSwap
21.14
21.25
21.07
RGB ↔ X
18.67
19.00
18.44
Material Fusion
23.19
25.68
21.44
Table 1. Quantitative comparison of Material Coating, measured by PSNR. Our CoatFusion outperforms all baselines across all metrics.
Figure 13. Ablation study. Latent albedo concatenation handles uniform albedo coatings (top row), but fails when a more complex geometry-aware wrapping is required. We show multiple ablation results to illustrate how this baseline fails to faithfully wrap the input albedo onto the geometry. In contrast, our proposed design plausibly maps textured coatings onto the object geometry. The no cyclic loss ablation performs well qualitatively, but yields a small but consistent drop in PSNR across all evaluation subsets. Three blocks, each showing the input triptych of albedo, mask and source image above outputs labelled Concat Albedo, No Cyclic Loss and Ours. The first block holds two compact examples: a hammer whose handle is coated cyan, and a stone guardian lion coated purple. In the second block a striped teal albedo is applied to the walls of a room corner: the Concat Albedo variants produce swirling marbled smears with no recognizable stripes, while No Cyclic Loss and Ours both wrap straight horizontal stripes correctly around the corner. In the third block a blue watercolor albedo is applied to a white jug on a shelf: the Concat Albedo variants invent bold teal and orange patterns, while Ours applies a soft blue wash that preserves the jug's shading and handle.
Figure 14. Human evaluation task example. Workers were given input and albedo images, and were tasked with choosing the most convincing material application result between CoatFusion and a baseline for multiple questions, one per material property. E.g. “Which result looks like it was coated with a metallic coating?” The layout shown to crowd workers. On the left, the source photograph of a dark metal desk on cobblestones and, beneath it, a green and cream floral albedo tile. On the right, two candidate results labelled A and B. In A the floral pattern follows the desk's own geometry, breaking around the drawer fronts and the recessed knee hole. In B the pattern is applied as though a printed cloth had been draped over the desk, hiding the drawers.
Figure 15. Human evaluation results. Top left: Comparison against baselines, where each bar represents our aggregate win rate against that specific baseline (e.g., roughly 95% against ZeST). Top right: Breakdown across material properties. Bottom: Win rate against each baseline, broken down by material property. ConcatAlbedo refers to our ablation baseline. Three bar charts of win rates, all with a red dashed line marking 50 percent. The first plots our aggregate win rate against each baseline: Blend If 65, ZeST 96, MatSwap 91, MARBLE 83, Color Blend 79, RGB to X 87, ConcatAlbedo 66, and MaterialFusion 90 percent. The second plots the win rate per material property: Metallic 84, Roughness 82, Texture 92, Thickness 75, and Translucent 97 percent. The third breaks the win rate down by baseline and property together, with five colored bars per baseline. Every bar in the first two charts lies above the 50 percent line; in the breakdown the only bar clearly below it is ConcatAlbedo on Metallic at about 18 percent, and the lowest remaining bars are Blend If on Texture and Thickness at roughly 55 percent.
Figure 16. Human evaluation failure case. An example where CoatFusion did not achieve a clear advantage over the baselines. For this specific combination of a rough, thin, and opaque non-metallic coating, our model reached a win rate of only 40% in both Roughness and Thickness categories. At the top, the inputs: a rendered grey desk on a wooden floor and a solid yellow-green albedo, with the requested traits given as a thin matte dielectric coating. Below, six results labelled Ours, Albedo Aligned ablation, RGB to X, Photoshop Color Blend, Photoshop Blend If, and MatSwap. All six render the desk in a similar yellow-green, which is why the comparison is inconclusive; RGB to X additionally covers the tabletop with speckled artifacts, and MatSwap leaves the drawer units uncoated and dark.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17. Data generation . We generate a synthetic dataset by randomly constructing scenes in Blender, rendering the original object appearance, and then generating multiple coated variants with varying coating parameters and masks (see main paper). A left-to-right pipeline diagram. On the left, the randomized inputs: a material properties panel listing roughness 0.68, transmission 0, metallic 0, thickness 0.8 and spherical UV mapping; a dark red and cream damask albedo with its blotchy binary coat mask; a screenshot of the coating material's Blender shader node graph; and thumbnail strips labelled 400 HDRIs and 105 objects together with icons for the light and camera setups. Arrows from all of these feed an orange trapezoid labelled Blender Cycles, which fans out on the right to four outputs: the albedo UV mapping, the projected coat mask, the coated render of an animal skull sculpture, and the original uncoated render of the same skull.
Figure 18. Example of a user study question of an object with a uniform coating. A user study question. On the left, the source photograph of a faded red propane cylinder on a wooden table, with a solid green albedo below it. On the right, two candidate results labelled A and B. In A the cylinder is evenly coated in green while its valve, carry handle and welded seam stay legible. In B the green is applied as a coarse, crusted layer that swells past the cylinder's silhouette and buries the handle.
Figure 19. Example of a user study question of an object with a uniform, transparent coating. A user study question. On the left, the source render of a dark wooden beam lying on a green-tinted planked floor, labelled TRANSPARENT. On the right, two candidate results labelled A and B. In A the beam is slightly darkened and glossier, with its grain still continuous along the length. In B the beam is covered in blotchy dark and orange mottling that breaks up the grain.
Figure 20. Comparison against MARBLE. Three examples in rows and three columns: the input, our result, and MARBLE. Each input cell holds the source photograph, the requested traits as text, the albedo swatch and the mask. Row one applies a thick translucent varnish to the doors of a wooden sideboard; ours darkens the two door panels while MARBLE repaints the entire cabinet grey and alters the objects standing on it. Row two applies a thick matte glitter paint to a stack of four ceramic cups; ours keeps the cups' original cream color and adds blue flecks, while MARBLE tints the whole stack pale blue and enlarges the speckles. Row three applies a thin green metallic coating to the rightmost of four wooden pawns; ours coats only that pawn, while MARBLE turns all four into translucent green glass.
Figure 21. Complementary Task. Example of removing and unwrapping the albedo from the masked geometry (1st row) and coating new geometries with the unwrapped albedo (2nd row). Two rows. The top row, coating removal, shows an input mask and a photograph of a mug printed with a cartoon scene of two characters by a campfire under a pink sky; the outputs are the same mug stripped back to plain dark grey ceramic and the recovered flat albedo of the scene, drawn in a dashed border. The bottom row applies that recovered albedo to two new objects: a plain terracotta vase becomes a vase carrying the campfire scene wrapped around its curvature, and a plain cardboard box becomes a box with the same scene continuing across its lid and side.
Figure 22. Training and architecture . Given an input image, albedo texture, user mask, and material traits, CoatFusion generates a coated result using a triptych-based diffusion formulation. During training, decomposition and composition are jointly optimized through cycle-consistent supervision. A two-panel diagram. The left panel, Data, lists the material properties, roughness 0.68, transmission 0, metallic 0, thickness 0.8, spherical UV mapping, together with a task type of add or remove coat, and shows the training triptych: the original render of an animal skull sculpture, a dark red and cream damask albedo, and the coated render. The right panel, In-Context Composition, feeds the masked image and its binary mask into a block labelled Flux Fill with LoRA, marked with a snowflake for the frozen backbone and a flame for the trained adapter, together with material trait embeddings that are also marked as trained. An arrow labelled Output leads to the completed triptych.
Figure 23. Additional Results. Our method generalizes to various objects and materials. Top: A watering can coated with glossy and matte green finishes. Bottom: A water bottle featuring patterned coatings with varying material traits. Two blocks, each showing the material parameters and albedo on the left and the source photograph beside two results. In the first block, glossy versus matte dielectric paint, a galvanized metal watering can held over white flowers is coated in solid green: the glossy result keeps bright specular highlights along the rim and spout, while the matte result is uniformly flat. In the second block, glossy dielectric versus metallic, a plain pink water bottle held against the sky is coated with a confetti speckle pattern: the dielectric result reads as painted enamel, while the metallic result gives the same speckles a reflective sheen and a darker, mirror-like ground.
Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose DealMaTe, using depth, normal, and lighting images for material transfer. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.
Nisha Huang, Yizhou Lin, Jie Guo +3
Tsinghua University, China · Pengcheng Laboratory, China · National Cheng-Kung University, Taiwan +2
This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa
Shangzhan Zhang, Sida Peng, Tao Xu +7
Zhejiang University · Ant Group · Shenzhen University
Recent diffusion-based methods for material transfer rely on image fine-tuning or complex architectures with assistive networks, but face challenges including text dependency, extra computational costs, and feature misalignment. To address these limitations, we propose MaTe, a streamlined diffusion framework that eliminates textual guidance and reference networks. MaTe integrates input images at the token level, enabling unified processing via multi-modal attention in a shared latent space. This design removes the need for additional adapters, ControlNet, inversion sampling, or model fine-tuning. Extensive experiments demonstrate that MaTe achieves high-quality material generation under a zero-shot, training-free paradigm. It outperforms state-of-the-art methods in both visual quality and efficiency while preserving precise detail alignment, significantly simplifying inference prerequisites.
Nisha Huang, Henglin Liu, Yizhou Lin +5
Tsinghua University · PengCheng Laboratory · Lenovo Research +1