EffectMaker: Unifying Reasoning and Generation for Customized Visual Effect Creation
Organizations: Tencent Hunyuan · City University of Hong Kong
Abstract
Visual effects (VFX) are essential for enhancing the expressiveness and creativity of video content, yet producing high-quality effects typically requires expert knowledge and costly production pipelines. Existing AIGC systems face significant challenges in VFX generation due to the scarcity of effect-specific data and the inherent difficulty of modeling supernatural or stylized effects. Moreover, these approaches often require per-effect fine-tuning, which severely limits their scalability and generalization to novel VFX. In this work, we present EffectMaker, a unified reasoning-generation framework that enables reference-based VFX customization. EffectMaker employs a multimodal large language model to interpret high-level effect semantics and reason about how they should adapt to a target subject, while a diffusion transformer leverages in-context learning to capture fine-grained visual cues from reference videos. These two components form a semantic-visual dual-path guidance mechanism that enables accurate, controllable, and effect-consistent synthesis without per-effect fine-tuning. Furthermore, we construct EffectData, the largest high-quality synthetic dataset containing 130k videos across 3k VFX categories, to improve generalization and scalability. Experiments show that EffectMaker achieves superior visual quality and effect consistency over state-of-the-art baselines, offering a scalable and flexible paradigm for customized VFX generation. Project page: https://effectmaker.github.io
Figures & tables
| Dataset | # Classes | # Videos | Label | Caption | Instruction |
|---|---|---|---|---|---|
| VFX-307 [ 15 ] | – | 190 | ✗ | ✗ | ✗ |
| Open-VFX [ 32 ] | 52 | 634 | ✓ | ✗ | ✗ |
| Omni-VFX [ 34 ] | 55 | 1700 | ✓ | ✗ | ✗ |
| HiggisField [ 17 ] | 145 | 1060 | ✓ | ✓ | ✗ |
| VAP-Data [ 2 ] | 100 | 90k | ✓ | ✗ | ✗ |
| EffectData (Ours) | 3000 | 130k | ✓ | ✓ | ✓ |
| Metrics | Methods | Cake-ify | Crumble | Decapitate | Deflate | Dissolve | Explode | Eye-pop | Harley | Inflate | Levitate | Melt | Squish | Ta-da | Venom | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VQ ↑ | VFX-Creator | 2.71 | 2.41 | 2.68 | 2.33 | 1.69 | 1.38 | 2.13 | 0.97 | 2.41 | 2.50 | 1.96 | 1.77 | 2.28 | 1.62 | 2.06 |
| Omni-Effect | 3.11 | 2.49 | 2.62 | 2.39 | 1.93 | 1.84 | 2.23 | 1.48 | 2.55 | 2.27 | 2.20 | 2.27 | 2.59 | 1.82 | 2.27 | |
| Wan2.2-FT | 2.15 | 1.42 | 2.63 | 2.15 | 1.64 | 0.91 | 2.23 | 1.71 | 1.89 | 2.12 | 1.74 | 2.11 | 2.04 | 1.69 | 1.89 | |
| Ours | 3.39 | 2.94 | 3.01 | 2.99 | 1.93 | 2.23 | 2.44 | 2.17 | 3.19 | 3.70 | 2.53 | 2.79 | 3.99 | 2.46 | 2.84 | |
| MQ ↑ | VFX-Creator | -0.05 | -0.15 | -0.05 | 0.13 | -0.17 | -0.35 | 0.11 | -0.26 | -0.14 | 0.48 | -0.16 | -0.41 | 0.77 | -0.21 | -0.03 |
| Omni-Effect | 0.12 | -0.27 | -0.08 | 0.02 | -0.09 | -0.33 | 0.04 | -0.28 | -0.11 | -0.04 | -0.14 | -0.37 | -0.09 | -0.34 | -0.14 |
| Conditioning | VQ | MQ | TA | CAS | RAS |
|---|---|---|---|---|---|
| Semantic | 2.78 | 0.16 | 1.06 | 4.20 | 3.84 |
| Visual | 2.48 | 0.12 | -0.38 | 2.24 | 1.48 |
| Semantic+Visual | 2.92 | 0.21 | 1.24 | 4.40 | 4.16 |
| # VFX Class | VQ | MQ | TA | CAS | RAS |
|---|---|---|---|---|---|
| 100 | 2.76 | 0.13 | 0.94 | 3.76 | 3.22 |
| 1000 | 2.89 | 0.19 | 1.21 | 4.22 | 4.04 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Controllable Video Generation
Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.
See all 300 papers