MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection
Organizations: Vector Institute for Artificial Intelligence Toronto, Canada
Abstract
Generalizing deepfake detection across facial manipulation and fully generated video remains challenging due to forgery traces varying with the manipulation method and scene content. Explicit spatial and temporal feature differences offer a way to capture this variation, but also respond to natural texture and motion, motivating their joint interpretation with appearance features. We investigate whether combining multiscale feature differences with adapted appearance representations improves detection across both forgery categories. We introduce Multiscale Appearance-Residual Streams (MARS), a two-path video detector. The residual path applies fixed spatial and temporal difference-of-Gaussians operators to frozen intermediate features and models the resulting differences alongside appearance context across frames. The semantic path adapts the encoder's normalization parameters for frame classification, and a clip-dependent gate combines their predictions. Under a shared protocol across seven datasets, MARS achieves the highest average AUROC of 91.43%, outperforming seven compared detectors. When trained on facial manipulations alone, MARS achieves 93.99% AUROC on GenVideo and 75.50% on DF26, demonstrating transfer to generated videos without generated-video training data. MARS also provides signed patch and frame contributions, whose faithfulness we evaluate through controlled spatial and temporal removal against random controls. MARS is publicly available for research: Project page: https://vectorinstitute.github.io/mars/ Github: https://github.com/VectorInstitute/mars
Figures & tables
| Train | Source test | Facial manipulation | Generated video | Avg. | ||||||||||||||
| FF++ | GTA-V | FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | ||||||||||
| Method | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | ||
| Frame-level | ||||||||||||||||||
| ✓ | ✗ | ForAda [ 8 ] | 86.67 | 21.61 | 86.66 | 22.28 | 96.67 | 9.53 | 95.29 | 8.16 | 71.26 | 35.00 | 54.54 | 46.59 | 39.53 | 57.60 | 75.80 | 28.68 |
| ✓ | ✗ | Effort [ 42 ] | 72.28 | 34.46 | 59.50 | 40.95 | 82.19 | 24.69 | 94.00 | 17.35 | 58.28 | 44.29 | 51.10 | 49.28 | 36.58 | 57.63 | 64.85 | 38.38 |
| ✓ | ✗ | GenD [ 44 ] | 70.42 | 34.55 | 66.13 | 39.50 | 98.91 | 5.08 | 51.10 | 53.06 | 38.76 | 56.07 | 64.32 | 40.55 | 59.04 | 43.51 | 64.10 | 38.90 |
| GenVideo | DF26 | |||||||||||||||||||
| Commercial | Open-source | |||||||||||||||||||
| Method | Sora | Morph Studio | Runway Gen2 | HotShot | Lavie | Show-1 | Moon Valley | Crafter | Model Scope | Wild Scrape | Grok Imagine | Kling 3.0 | Veo 3.1 | Wan 2.6 | HV1.5 I2V | HV1.5 T2V | Wan2.2 I2V | Wan2.2 T2V | LTX2.3 I2V | LTX2.3 T2V |
| Frame-level | ||||||||||||||||||||
| ForAda | 59.05 | 73.09 | 46.58 | 72.77 | 65.08 | 65.49 | 57.70 | 66.61 | 79.72 | 70.96 | 19.68 | 30.60 | 14.16 | 30.90 | 40.99 | 58.11 | 60.62 | 42.36 | 53.38 | 32.36 |
| Effort | 46.47 | 63.11 | 47.72 | 58.61 | 55.66 | 61.04 | 47.06 | 52.52 | 69.30 | 67.16 | 26.74 | 44.56 | 20.91 | 30.30 | 45.87 | 42.59 | 49.79 | 36.69 | 47.47 | 27.08 |
| GenD | 66.36 | 61.41 | 76.76 | 43.09 | 58.02 | 47.56 | 73.53 | 78.56 | 63.36 | 64.50 | 78.63 | 74.64 | 52.40 | 58.52 | 51.14 | 63.36 | 53.34 | 50.10 | 60.23 | 60.85 |
| Path | Residual-path features | Norm | AUROC | |||||||
| Variant | Sem. | Res. | App. | Spat. | Temp. | FD | LN | F | G | Avg. |
| MARS (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 92.95 | 84.63 | 91.43 |
| Residual path only | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | – | 89.13 | 69.50 | 84.80 |
| Semantic path only | ✓ | ✗ | – | – | – | – | ✓ | 89.89 | 87.14 | 90.24 |
| w/o appearance | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | 93.69 | 82.56 | 91.28 |
| w/o spatial DoG | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | 85.54 | 73.00 | 83.96 |
| Method | FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | Mean |
| Frame-level | ||||||||
| ForAda | 86.40 0.62 | 84.05 1.65 | 95.37 1.30 | 95.74 0.56 | 72.19 0.87 | 68.30 2.69 | 40.37 0.72 | 77.49 0.31 |
| Effort | 71.39 0.46 | 60.54 2.89 | 87.52 3.21 | 91.48 0.87 | 57.49 1.05 | 51.74 9.04 | 38.60 3.54 | 65.54 1.98 |
| GenD | 70.37 7.34 | 68.17 4.63 | 98.44 1.32 | 53.15 15.51 | 48.91 11.60 | 63.26 3.53 | 61.51 3.09 | 66.26 5.62 |
| DFD-HR | 99.75 0.06 | 93.45 0.65 | 99.28 0.37 | 97.30 1.49 | 85.07 2.66 | 80.41 4.25 | 71.78 3.26 | 89.58 0.39 |
| Clip-level | ||||||||
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
| Encoder | |
| Backbone | PE-Core-L/14, , frozen |
| Patch grid, width | ( ), |
| Blocks, split block | , |
| Residual path | |
| Projection width | |
| Dataset | Split / use | Fake content | Real | Fake | Resolution | FPS | Duration (s) | Median frames |
| FF++ c23 | Train | Facial manipulation | 720 | 2,880 | 480p | 25 | 17 | 414 |
| FF++ c23 | Validation | Facial manipulation | 140 | 560 | 480p | 30 | 13 | 398 |
| FF++ c23 | Test | Facial manipulation | 140 | 560 | 480p | 25 | 18 | 444 |
| GTA-V | Train | Rendered scenes | 0 | 141 | – | – | 336 | |
| GTA-V | Test | Rendered scenes | 0 | 30 | – | – | 389 | |
| CelebDF-V2 | Test | Facial manipulation | 178 | 340 | 480p | 30 | 12 | 372 |
| Method | Encoder | Trained components | Input | Clip aggregation | Objective | Trainable |
| UNITE | SigLIP-so400m (frozen) | Temporal transformer, head | 384 | Temporal transformer | CE + AD | 63.91M |
| GenD | PE-Core-L | Normalization layers, head | 336 | Mean of frame logits | CE + uniformity + alignment | 0.11M |
| ForAda | CLIP ViT-L/14 (frozen) | Side adapter, queries, head | 224 / 256 | Mean of frame logits | CE (mask-based terms omitted) | 5.98M |
| DFD-HR | CLIP ViT-L/14 (frozen) | MoE adapters, token routers, head | 448 | Mean of frame logits | CE + ranking | 103.48M |
| DeMamba | X-CLIP B/16 | Full encoder, Mamba, head | 224 | Clip-level | BCE | 125.70M |
| Effort | CLIP ViT-L/14 (frozen) | Rank-1 SVD residuals, head | 224 | Mean of frame logits | CE + orthogonality | 0.20M |
| Encoder | Split block | F | G | Avg. |
| Split layer | ||||
| PE | 6 | 89.97 | 81.04 | 88.58 |
| PE | 12 | 92.95 | 84.63 | 91.43 |
| PE | 18 | 94.06 | 77.04 | 89.89 |
| PE | 24 | 93.61 | 80.80 | 90.68 |
| Backbone (split at block 12) | ||||
| 3 epochs | 5 epochs | 10 epochs | ||||
| Method | F | G | F | G | F | G |
| Frame-level | ||||||
| ForAda | 87.72 | 52.37 | 85.76 | 53.31 | 84.79 | 53.14 |
| Effort | 73.01 | 47.21 | 74.26 | 47.16 | 75.23 | 48.05 |
| GenD | 66.35 | 62.27 | 69.40 | 77.67 | 88.15 | 84.96 |
| DFD-HR | 94.00 | 77.21 | 94.81 | 77.29 | 94.78 | 75.58 |
| GenD | DeMamba | MARS | ||
| Dataset | CE only, 3 ep | lr , 3 ep | lr , 10 ep | Semantic path only |
| Source test | ||||
| FF++ | 98.00 | 53.21 | 83.04 | 95.67 4.67 |
| Facial manipulation | ||||
| CelebDF | 90.70 | 52.90 | 70.18 | 83.83 11.87 |
| DF-TIMIT | 99.40 | 49.81 | 93.19 | 99.42 0.13 |
| Fusion weight | F | G | Avg. | EER |
| Fixed | ||||
| 0 (residual path) | 90.62 | 76.21 | 87.66 | 18.39 |
| 0.25 | 92.44 | 81.69 | 90.33 | 15.32 |
| 0.5 (equal average) | 92.99 | 84.25 | 91.35 | 13.65 |
| 0.75 | 92.57 | 84.72 | 91.19 | 14.17 |
| 1 (semantic path) | 89.41 | 83.78 | 89.00 | 15.44 |
| Residual streams | Scales | F | G | Avg. |
| Streams and scales | ||||
| 7 | 92.95 | 84.63 | 91.43 | |
| 3 | 90.58 | 79.57 | 88.68 | |
| None (appearance only) | – | 86.23 | 90.56 | 88.91 |
| Group | Clips | Frame AUROC | Top-1 | Chance |
| Overall | 3,360 | 80.73 | 93.30 | 37.50 |
| Segment length | ||||
| 16 frames | 1,680 | 79.15 | 90.06 | 25.00 |
| 32 frames | 1,680 | 82.30 | 96.55 | 50.00 |
| Segment position | ||||
| Start | 1,120 | 79.65 | 92.68 | 37.50 |
| Semantic frames | FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | Avg. |
| 8 | 99.01 / 4.37 | 91.05 / 16.81 | 99.87 / 1.41 | 97.38 / 12.24 | 84.49 / 20.71 | 92.53 / 14.10 | 70.96 / 34.02 | 90.76 / 14.81 |
| 16 (default) | 98.98 / 4.46 | 91.22 / 14.80 | 99.95 / 0.94 | 96.58 / 8.16 | 84.05 / 22.50 | 93.44 / 12.20 | 75.81 / 29.78 | 91.43 / 13.26 |
| 32 | 99.26 / 4.20 | 91.43 / 15.23 | 99.88 / 1.80 | 97.25 / 12.24 | 85.97 / 21.43 | 92.82 / 14.19 | 75.13 / 30.69 | 91.68 / 14.25 |
| 64 | 99.18 / 5.09 | 91.56 / 16.09 | 99.91 / 1.72 | 97.42 / 13.27 | 85.92 / 20.71 | 93.41 / 12.63 | 76.92 / 30.81 | 92.05 / 14.33 |
| Train | Source test | Facial manipulation | Generated video | Avg. | ||||||
| FF++ | GTA-V | Method | FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | |
| Frame-level | ||||||||||
| ✓ | ✗ | ForAda [ 8 ] | 96.46 | 92.37 | 98.48 | 96.70 | 73.77 | 52.99 | 87.63 | 85.48 |
| ✓ | ✗ | Effort [ 42 ] | 91.87 | 73.95 | 90.90 | 94.50 | 59.48 | 52.61 | 86.40 | 78.53 |
| ✓ | ✗ | GenD [ 44 ] | 89.47 | 77.67 | 99.44 | 49.48 | 44.90 | 67.39 | 92.33 | 74.38 |
| ✓ | ✗ | DFD-HR [ 37 ] | 99.95 | 97.20 | 99.98 | 97.98 | 88.77 | 67.27 | 94.96 | 92.30 |
| GenVideo | DF26 | |||||||||||||||||||
| Commercial | Open-source | |||||||||||||||||||
| Method | Sora | Morph Studio | Runway Gen2 | HotShot | Lavie | Show-1 | Moon Valley | Crafter | Model Scope | Wild Scrape | Grok Imagine | Kling 3.0 | Veo 3.1 | Wan 2.6 | HV1.5 I2V | HV1.5 T2V | Wan2.2 I2V | Wan2.2 T2V | LTX2.3 I2V | LTX2.3 T2V |
| Frame-level | ||||||||||||||||||||
| ForAda | 40.84 | 32.38 | 53.20 | 32.39 | 39.20 | 38.01 | 43.58 | 36.71 | 27.11 | 33.40 | 70.67 | 62.43 | 76.26 | 63.09 | 54.43 | 43.17 | 41.33 | 53.51 | 47.42 | 61.99 |
| Effort | 51.78 | 39.99 | 51.52 | 43.28 | 46.21 | 42.15 | 51.67 | 47.78 | 35.46 | 36.75 | 66.38 | 52.90 | 70.92 | 64.89 | 53.14 | 56.09 | 51.48 | 57.20 | 51.11 | 65.68 |
| GenD | 39.26 | 42.57 | 30.83 | 55.75 | 44.22 | 51.88 | 32.90 | 28.61 | 41.15 | 40.58 | 27.91 | 30.72 | 49.04 | 42.10 | 50.55 | 41.70 | 47.60 | 50.74 | 44.46 | 43.54 |
| Source test | Facial manipulation | Generated video | Avg. | ||||||||||||||
| FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | |||||||||||
| Variant | Params | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER | AUROC | EER |
| Full MARS | 67.54 | 98.98 | 4.46 | 91.22 | 14.80 | 99.95 | 0.94 | 96.58 | 8.16 | 84.05 | 22.50 | 93.44 | 12.20 | 75.81 | 29.78 | 91.43 | 13.26 |
| Paths | |||||||||||||||||
| Residual path only | 67.43 | 98.07 | 7.05 | 82.42 | 25.29 | 99.26 | 4.22 | 97.79 | 6.12 | 77.04 | 27.86 | 74.76 | 31.45 | 64.23 | 40.13 | 84.80 | 20.30 |
| Semantic path only | 0.11 | 97.83 | 7.32 | 89.16 | 18.67 | 99.27 | 2.66 | 83.80 | 12.24 | 87.34 | 20.00 | 93.59 | 14.25 | 80.68 | 26.59 | 90.24 | 14.53 |
| Setting | Steps | FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | Avg. |
| MARS (Ours): default | 1,449 | 98.98 / 4.46 | 91.22 / 14.80 | 99.95 / 0.94 | 96.58 / 8.16 | 84.05 / 22.50 | 93.44 / 12.20 | 75.81 / 29.78 | 91.43 / 13.26 |
| Training duration (64-frame clips) | |||||||||
| 5 epochs | 2,415 | 99.62 / 2.86 | 92.29 / 14.08 | 99.90 / 1.33 | 96.58 / 8.16 | 86.88 / 18.57 | 92.26 / 13.08 | 74.38 / 31.32 | 91.70 / 12.77 |
| 10 epochs | 4,830 | 99.38 / 4.29 | 90.81 / 15.81 | 99.91 / 1.41 | 96.00 / 8.16 | 84.71 / 21.79 | 95.31 / 9.71 | 74.75 / 31.10 | 91.55 / 13.18 |
| Clip length (three epochs) | |||||||||
| 8 frames | 10,410 | 99.40 / 3.57 | 92.55 / 14.51 | 99.91 / 1.56 | 95.84 / 10.20 | 86.30 / 21.43 | 94.44 / 11.80 | 79.11 / 27.25 | 92.50 / 12.90 |
| Method | FF++ | CelebDF | DF-TIMIT | UADFV | HiFiFace | GenVideo | DF26 | Mean | ||||||||||||||||
| AUC | EER | AP | AUC | EER | AP | AUC | EER | AP | AUC | EER | AP | AUC | EER | AP | AUC | EER | AP | AUC | EER | AP | AUC | EER | AP | |
| Frame-level | ||||||||||||||||||||||||
| ForAda | 82.35 | 25.94 | 95.06 | 78.28 | 28.08 | 86.64 | 89.64 | 17.68 | 94.29 | 92.03 | 13.04 | 93.65 | 70.30 | 35.95 | 69.61 | 64.31 | 39.19 | 60.03 | 38.45 | 57.20 | 87.09 | 73.62 | 31.01 | 83.77 |
| Effort | 69.75 | 36.25 | 90.84 | 56.77 | 44.57 | 71.78 | 73.33 | 32.93 | 84.87 | 83.93 | 23.13 | 84.54 | 58.00 | 44.58 | 58.75 | 58.30 | 43.94 | 58.01 | 37.29 | 58.65 | 86.49 | 62.48 | 40.58 | 76.47 |
| GenD | 61.77 | 41.26 | 85.96 | 59.18 | 43.62 | 72.71 | 74.13 | 31.45 | 84.56 | 63.54 | 40.08 | 62.06 | 49.90 | 48.77 | 51.48 | 68.54 | 36.31 | 70.26 | 46.96 | 52.64 | 89.10 | 60.57 | 42.02 | 73.74 |
| DFD-HR | 90.60 | 16.18 | 97.37 | 80.57 | 27.06 | 88.27 | 87.82 | 18.30 | 93.13 | 86.58 | 17.63 | 86.00 | 64.60 | 38.77 | 66.44 | 62.36 | 41.04 | 64.47 | 48.74 | 50.61 | 89.68 | 74.47 | 29.94 | 83.62 |
| Training | Testing | ||||||
| Method | GPU-h | kWh | kg CO 2 e | Water (L) | GPU-h | kWh | kg CO 2 e |
| Frame-level | |||||||
| ForAda | 27.4 | 8.13 | 0.526 | 14.6–32.5 | 4.2 | 1.22 | 0.079 |
| Effort | 14.2 | 3.96 | 0.256 | 7.1–15.8 | 3.3 | 0.73 | 0.047 |
| GenD | 25.5 | 7.48 | 0.484 | 13.5–29.9 | 4.3 | 1.15 | 0.075 |
| DFD-HR | 86.1 | 24.80 | 1.605 | 44.6–99.2 | 9.3 | 2.75 | 0.178 |
Deepfake Detection
Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.
See all 264 papers