cs.CVSep 29, 2026

AESOP: Asymmetric Human-Camera Generation with Translation-Intensity Control

Authors: Jingzhong Lin, Zhanke Wang, Heng Li, Wenxiang Liu, Zhao Zhang, Kecheng Tang, Dongdong Xiang, Changbo Wang, +4 more

Organizations: East China Normal University · Peking University · Sun Yat-sen University · Tencent

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human motion defines an action, while a camera trajectory determines how it is presented. Camera generation for a given human motion and joint human-camera generation are usually treated as separate tasks, although both share an asymmetric dependency: human motion can be generated independently, whereas the camera responds to the realized action. We introduce AESOP, a unified framework with an independent human pathway and a shared human-conditioned camera generator. Its asymmetric architecture serves both tasks while preserving the human output during camera generation. Although human context anchors the shot to the action and camera text describes its movement, translation intensity remains underspecified. We therefore construct trajectory pairs that differ in camera translation magnitude while sharing human motion and camera text, then use these pairs to learn an explicit intensity condition. Experiments on the PulpMotion dataset demonstrate strong camera distributional and framing quality in both tasks and effective control over camera translation intensity.

Figures & tables

Figure 1: AESOP supports (a) camera generation for a given human motion, (b) joint human–camera generation, and (c) continuous camera translation-intensity control (the amount of camera travel) in both tasks without changing the supplied or generated human motion.
Figure 2: AESOP overview. (A) Separate encoders produce 128-channel human and 64-channel camera latents; the camera decoder reads their concatenation. (B,C) Human generation supplies the complete latent sequence to both human decoding and camera conditioning. Given-human generation instead encodes the supplied motion. (D,E) Human and camera Transformer blocks.
Figure 3: Training target construction of DPA and IPA. (a,b) Opposite directions share human motion and the initial camera state. (c,d) Intensity variants retain human motion and camera text.
 camera distribution & alignmentcamera framing & geometryhuman distribution & alignment
MethodFDC ↓\downarrowCLaTr ↑\uparrowF1 ↑\uparrowRecall C ↑\uparrowr-FPD ↓\downarrowOut ↓\downarrowADE ↓\downarrowRot. ↓\downarrowFD TMR ↓\downarrowTMR ↑\uparrow
GT reference≈0\approx 070.2370.9451.000.0030.710.0000.002≈0\approx 018.398
PulpMotion AE (sym.)16.59 ±\pm 3.5259.91 ±\pm 1.380.739 ±\pm 0.0320.973 ±\pm 0.0050.133 ±\pm 0.023.93 ±\pm 0.550.110 ±\pm 0.011.37 ±\pm 0.1038.79 ±\pm 9.9920.15 ±\pm 0.28
AESOP AE (asym.)0.11 ±\pm 0.1070.10 ±\pm 0.130.941 ±\pm 0.0010.999 ±\pm 0.00040.089 ±\pm 0.012.93 ±\pm 0.280.029 ±\pm 0.00060.41 ±\pm 0.0110.05 ±\pm 0.8917.56 ±\pm 0.12
Given-human camera generation
DanceCamera3D180.08 ±\pm 5.2420.92 ±\pm 0.540.252 ±\pm 0.0020.600 ±\pm 0.012.692 ±\pm 0.4821.72 ±\pm 2.332.995 ±\pm 0.0763.52 ±\pm 1.19––
Table 1: Quantitative comparison. AE rows report reconstruction; the remaining model rows report generation. AESOP (no aug.) uses original pairs, (+DPA) adds direction pairs, and full AESOP further adds intensity pairs. Human scores are shared across AESOP camera-training phases. Bold marks the best displayed value within each reconstruction/task group, excluding GT; blue marks AESOP.
Figure 4: Qualitative comparison on test examples. (a) Given-human camera generation with a pull-out prompt. (b) Joint human–camera generation with a push-in prompt. Each method has a global view and three selected camera projections, ordered from top to bottom in time. Global views show human–camera geometry at individually fitted scales; projections show framing over time.
MethodPrecision ↑\uparrowRecall ↑\uparrowDensity ↑\uparrowCoverage ↑\uparrow
PulpMotion DiT0.809 ±\pm 0.020.215 ±\pm 0.020.748 ±\pm 0.050.465 ±\pm 0.02
PulpMotion MAR0.802 ±\pm 0.0050.312 ±\pm 0.010.705 ±\pm 0.020.504 ±\pm 0.002
AESOP0.879 ±\pm 0.0020.835 ±\pm 0.010.933 ±\pm 0.020.785 ±\pm 0.01
Table 2: Human distributional fidelity and coverage in joint generation in the common frozen human embedding space. Bold marks the best value in each column.
Figure 5: Translation-intensity examples with shared human context, text and camera noise. Outputs are regenerated at each intensity; numbers report first-to-last camera-center displacement. Fading snapshots indicate temporal order. Global views are fitted separately to each trajectory's spatial extent.
Compared withPref.95% CI
Given-human
CCD75.3[64.8, 85.5]
DanceCamera3D99.2[97.5, 100.0]
DIRECTOR-C86.6[81.5, 91.5]
Joint
PulpMotion DiT86.3[80.1, 91.9]
Table 3: Overall-quality preference for AESOP (%). Scores average clip-level responses, with ties receiving half credit. Intervals are participant-cluster 95% CIs.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: human–camera representation dependencies. (a) PulpMotion’s symmetric representation encodes both inputs with a shared encoder, reconstructs each from its latent, and learns an auxiliary framing latent from their concatenation ( Courant et al., 2026 ) . (b) AESOP separately encodes the two inputs. The human decoder reads only human latents; the camera decoder concatenates camera latents with frozen human latents (circled C). Human representation training is completed before camera training.
 TrainingReconstruction
MethodIndependent human learningcamera-independent humanhuman-aware camera
PulpMotion (symmetric)×\times Both streams trained together×\times✓\checkmark
AESOP (asymmetric)✓\checkmark Train human first, then freeze✓\checkmark✓\checkmark
Appendix
Table S1: Training and reconstruction dependencies. Both representations provide human context for camera reconstruction. AESOP additionally separates human learning from camera training and makes human reconstruction independent of camera input.
 human reconstructioncamera reconstruction
AutoencoderGlobal MPJPE ↓\downarrowAligned MPJPE ↓\downarrowRoot ADE ↓\downarrowRoot FDE ↓\downarrowFDE ↓\downarrowTrans. vel. ↓\downarrowRot. vel. ↓\downarrowTrans. jerk ↓\downarrow
PulpMotion (sym.)0.1620 ±\pm 0.020.0617 ±\pm 0.010.1376 ±\pm 0.010.3283 ±\pm 0.040.1677 ±\pm 0.02124.28 ±\pm 3.0210.55 ±\pm 0.10154.67 ±\pm 1.54
AESOP (asym.)0.1304 ±\pm 0.010.0479 ±\pm 0.0030.1063 ±\pm 0.010.2611 ±\pm 0.020.0356 ±\pm 0.00210.11 ±\pm 0.8110.98 ±\pm 0.218.55 ±\pm 0.40
Appendix
Table S2: Autoencoder reconstruction on the test split. Sym./asym. denote PulpMotion's symmetric and AESOP's asymmetric autoencoders. Motion errors are clip-balanced and measured in mm/s, ∘ /s and m/s 3 , respectively. MPJPE, ADE and FDE are in meters; aligned MPJPE removes root translation. Bold marks the best value between the autoencoders.
StagePhaseUpdatesBatchLearning rateFrozen components
Reconstructionhuman autoencoder210K1285×10−55\times 10^{-5}None
 camera autoencoder210K1285×10−55\times 10^{-5}human autoencoder
Generationhuman flow105K1282×10−42\times 10^{-4}Autoencoders
 Original camera105K12810−410^{-4}Autoencoders, human flow
 camera + DPA35K1202×10−52\times 10^{-5}Autoencoders, human flow
 camera + DPA + IPA15K1202×10−52\times 10^{-5}Autoencoders, human flow
Appendix
Table S3: Training schedule. The final phase additionally trains the intensity MLP at 10−410^{-4} .
Figure S2: DPA construction and validation. Left: the original Dolly-in trajectory. Middle: its opposite Dolly-out target, with human motion, camera rotation, field of view and timing fixed. Right: a reconstruction rejected because its direction change exceeds the acceptance threshold after decoding. The gold point marks the initial camera center; global views show the starting human pose and full trajectory, and projections show the synchronized final frame. The three stages summarize source filtering, direction reversal and target checks; 601 sources pass the final reconstruction check.
Figure S3: IPA target construction. Reduced, original and increased translation share human motion and camera text. Camera travel changes from 0.356 to 0.634 to 0.786 m in this example. The gold point marks the initial camera center; global views and synchronized final-frame projections show the resulting geometry and framing. The three stages summarize the active branch, which supplies 16,000 training targets from 8,000 sources. The accompanying null branch supplies 2,000 low-translation sources with unchanged targets.
OperationAcceptance rules
DPA: direction pairs
Select source eventsDuration ≥45\geq 45 frames; same-sign velocity fraction ≥0.80\geq 0.80 ; signed net displacement/path ≥0.60\geq 0.60 . Truck displacement ≥0.20\geq 0.20 m; Dolly displacement ≥max⁡(0.20 m,0.10ρts)\geq\max(0.20\text{ m},0.10\rho_{t_{s}}) .
Check target geometryOpposite-target human distance >0.25>0.25 m; opposite/original response ∈[0.80,1.25]\in[0.80,1.25] . Rotation/FOV unchanged; Truck vertical shift ≤10−5\leq 10^{-5} m and radial relative change ≤0.10\leq 0.10 ; Dolly radial-direction change ≤0.05∘\leq 0.05^{\circ} .
Check framing and effectNo zero-visible frame; visibility ≥0.80\geq 0.80 ; opposite outscreen increase ≤0.05\leq 0.05 . Truck: projected center shift ≥0.08\geq 0.08 image width or viewpoint change ≥6∘\geq 6^{\circ} . Dolly: peak box-height ratio change ≥0.12\geq 0.12 .
Check reconstructed targetsFeature MSE ≤0.01181\leq 0.01181 ; both direction signs correct; decoded/raw response ratios ∈[0.70,1.30]\in[0.70,1.30] . No framing failure; Vdec≥1V_{\rm dec}\geq 1 and Vdec/Vraw≥0.70V_{\rm dec}/V_{\rm raw}\geq 0.70 .
IPA: intensity pairs
Appendix
Table S4: Construction acceptance criteria for DPA and IPA, grouped by source selection, target geometry, framing and reconstruction.
MethodInputs and conditioningSampling and guidance
DIRECTOR-CCamera text and human root trajectoryEDM: 10 Euler/Heun steps; camera CFG 1.4
DanceCamera3DCamera text and full human motion; text replaces musicDDIM: 50 steps, η=1\eta=1 ; human/camera CFG 1.75/1
CCDCamera text; text-only networkDDPM: 1,000 steps; camera CFG 2
CCD-HCamera text and human joints; text cross-attentionDDPM: 1,000 steps; camera CFG 2
PulpMotion DiTHuman and camera text; symmetric representationDDPM: 50 steps; (gc,gm,gz)=(11,−1,0)(g_{c},g_{m},g_{z})=(11,-1,0)
PulpMotion MARHuman and camera text; symmetric representation18 autoregressive iterations, each with 50 DDPM steps; (gc,gm,gz)=(3.5,2,0)(g_{c},g_{m},g_{z})=(3.5,2,0)
Appendix
Table S5: Baseline inputs, conditioning and sampling. For PulpMotion, the tuple denotes conditional-text guidance, autoregressive-context guidance, and auxiliary projection guidance, respectively. A negative autoregressive-context weight selects standard CFG; zero auxiliary projection weight disables that guidance term. MAR generates autoregressively, with a diffusion sampler inside each iteration. EDM, DDIM and DDPM follow Karras et al. (2022) , Song et al. (2021) and Ho et al. (2020) , respectively.
 Given-human generationJoint generation
CheckpointOriginal ↑\uparrowReversed ↑\uparrowCombined ↑\uparrowOriginal ↑\uparrowReversed ↑\uparrowCombined ↑\uparrow
Camera original86.38 ±\pm 0.8675.66 ±\pm 0.6881.02 ±\pm 0.1583.15 ±\pm 0.1376.42 ±\pm 0.4379.78 ±\pm 0.28
35K factual-pair control87.60 ±\pm 0.4377.71 ±\pm 0.1382.66 ±\pm 0.1784.95 ±\pm 1.1378.51 ±\pm 0.1781.73 ±\pm 0.51
+ DPA89.66 ±\pm 0.6779.00 ±\pm 0.2284.33 ±\pm 0.2886.63 ±\pm 0.7779.58 ±\pm 0.5083.11 ±\pm 0.14
Appendix
Table S6: Direction accuracy (%). DPA and factual continuations share the starting checkpoint, 35K updates, batch allocation and sampling schedule; factual pairs replace directional counterfactuals in the control. Bold marks the best mean in each column. Section D.2 defines paired sampling and scoring.
Figure S4: User-study interface. Anonymized A/B videos show the generated camera view above an external spatial view. Participants compare text alignment, motion quality, framing and task-specific overall quality. The joint task additionally evaluates human motion.
Figure S6: Camera generation conditioned on HumanML3D motions. Each example shows a global view and three successive camera views; lighter poses indicate earlier frames. Global views are fitted separately to each scene.
Figure S7: Joint generation from HumanML3D action descriptions. Each example shows generated human motion and camera trajectory in a global view and three successive camera views.
Figure S8: Camera generation conditioned on HY-Motion outputs. Four synthesized human motions are paired with static, pull-out and pull-in camera prompts. Each example includes a global view and three successive camera views.
Figure S9: Additional given-human camera generation examples. Global views are fitted separately to each trajectory.
Figure S10: Additional joint human–camera generation examples. Global views are fitted separately to each trajectory.

Explore similar work

CardsList

  1. Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation

    Jun 1, 2026Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra +4Cinematic IdealDramadirector

    Koç University · University College London, Adobe · Adobe +1

  2. UniMoCa: Unifying Motion and Camera Controls as Visual Proxies for Faithful Human Video Generation

    Aug 3, 2026Liming Tan, Ye Chen, Hao Zhang +3Camera ControlHuman Motion Generation

    Shanghai Jiao Tong University · Monash University · USC-SJTU Institute of Cultural and Creative Industry

  3. ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation

    May 7, 2026Omar El Khalifi, Thomas Rossi, Oscar Fossey +6Camera-Conditioned Video GenerationVideo Generation

    Kinetix, France · University of Oxford, United Kingdom · MBZUAI, United Arab Emirates