DynOSeg: Dynamic Object Segmentation via Motion-Geometry Fusion
Organizations: Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney, NSW, 2006, Australia
Abstract
Moving Object Segmentation (MOS) detects and separates dynamic objects from the static background in a video, which plays a fundamental role in dynamic-scene understanding. Most MOS methods rely on pure visual motion cues alone, such as optical flow or point tracks, which is inherently ambiguous due to the compounded camera and object motions as well as motion-induced appearance changes, e.g. shadows and reflections. Naive fusion of additional information such as depth and semantics, however, cannot leverage their complementarity to the fullest extent. We therefore propose DynOSeg, a novel MOS approach that takes on a two-stage fusion structure. DynOSeg first combines a geometry-aware residual map and depth encoding into a geometric representation of motion, delineating camera motions and object dynamics. We then further incorporate semantic features to refine object understanding. We have additionally curated a diverse training corpus including CARLA-MOS, a new synthetic urban-driving dataset for MOS training. Our experiments show that DynOSeg consistently outperforms published state-of-the-art MOS methods across a diverse range of datasets. Our code and pretrained models can be found at https://github.com/ACFR-RPG/DynOSeg
Figures & tables
| Dataset | Type | #Seq. | #Frames |
|---|---|---|---|
| Kubric [ 14 ] | Synthetic | 6,000 | 138k |
| Dynamic Replica [ 25 ] | Synthetic | 483 | 145k |
| OCLR-Syn [ 51 ] | Synthetic | 4,600 | 138k |
| CARLA-MOS | Synthetic | 1,000 | 300k |
| YTVOS-Moving [ 7 ] | Real-world | 395 | 10.2k |
| YTVOS-Moving ∗ | Real-world | 342 | 9.5k |
| Method | Year | DAVIS2017-M | SegTrackV2 | FBMS-59 | DSEC-MOS | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EM [ 30 ] | TPAMI’23 | 62.8 | 61.6 | 64.0 | 54.9 | 51.5 | 58.3 | 58.9 | 58.4 | 59.4 | 5.8 | 4.5 | 7.0 |
| OCLR-TTA [ 51 ] | NeurIPS’22 | 69.4 | 69.9 | 69.0 | 63.4 | 61.3 | 65.6 | 63.7 | 63.2 | 64.3 | 6.5 | 4.8 | 8.1 |
| ABR [ 52 ] | ECCV’24 | 72.3 | 72.7 | 72.0 | 72.4 | 71.1 | 73.7 | 75.0 | 75.7 | 74.3 | 6.3 | 3.5 | 9.2 |
| SegAnyMo [ 19 ] | CVPR’25 | 88.5 | 88.0 | 88.9 | 76.3 | 73.5 | 79.1 | 72.6 | 69.4 | 75.9 | 22.6 | 19.9 | 25.2 |
| GeoMotion [ 15 ] | CVPR’26 | 81.2 | 81.1 | 81.3 | 75.3 | 70.9 | 79.8 | 78.8 | 76.7 | 80.9 | 20.5 | 17.3 | 23.8 |
| Training setting | DAVIS2017-M | DSEC-MOS | ||
|---|---|---|---|---|
| w/ CARLA-MOS | 90.1 | 73.0 | ||
| w/o CARLA-MOS | 87.5 | 65.7 | ||
| Exp. | Setting | DAVIS2017-M | DSEC-MOS |
|---|---|---|---|
| (A) | Optical Flow + RGB + Depth ‡ | 86.0 | 63.6 |
| (B) | Residual Map only | 55.4 | 61.2 |
| (C) | (B) + Depth ‡ | 79.8 | 64.1 |
| (D) | (B) + RGB | 83.5 | 65.6 |
| (E) | (D) + Depth † | 78.6 | 63.3 |
| PlainCNN replacement | DAVIS2017-M | DSEC-MOS |
|---|---|---|
| Residual encoder only | 86.0 | 65.4 |
| Depth encoder only | 84.8 | 64.9 |
| Both encoders | 85.4 | 65.1 |
| None (DynOSeg) | 90.1 | 73.0 |
| Exp. | Fusion Structure | DAVIS2017-M | DSEC-MOS |
|---|---|---|---|
| (A) | Two-stage | 90.1 | 73.0 |
| (B) | Concat | 83.6 | 69.0 |
| (C) | Only GSF | 82.8 | 69.0 |
| (D) | Only GLSF | 84.5 | 70.6 |
| (E) | Swapped order | 83.7 | 69.5 |
| Pose source | Depth source | Setting | |
|---|---|---|---|
| KITTI OXTS (GT) | FoundationStereo | Reference | 89.0 |
| ViPE | ViPE | Offline | 84.0 |
| DynOSAM | DynOSAM | Online | 89.0 |
| Input resolution | DynOSeg | RAFT | Residual | |
|---|---|---|---|---|
| Wall-clock | GPU | GPU | GPU | |
| 20.13 | 19.00 | 25.93 | 0.43 | |
| 32.95 | 32.28 | 31.99 | 0.44 | |
| 46.93 | 47.15 | 44.89 | 0.46 | |
| 102.43 | 106.20 | 84.57 | 0.58 | |
| 252.69 | 270.65 | 173.47 | 1.02 | |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Conv. | Operation | Channels (in/out) | Kernel / stride |
|---|---|---|---|
| 1 | Conv–BN–SiLU | / | |
| 2 | Conv–BN–SiLU | / | |
| – | MaxPool | / | |
| 3 | Conv–BN–SiLU | / | |
| 4 | Conv–BN–SiLU | / | |
| – | LayerNorm2d | – |
| Encoder branch | DynOSeg | PlainCNN | Difference |
|---|---|---|---|
| Depth | 948,096 | 962,688 | |
| Residual | 946,466 | 960,960 |
| Item | Setting |
|---|---|
| Dataset source | KITTI Raw sequences with tracklet annotations |
| Camera stream | Left color camera, image_02 |
| Number of sequences / frames | 38 sequences / 12,915 frames |
| Reference camera poses | KITTI GT camera poses derived from OXTS and calibration |
| Reference dense depth | FoundationStereo applied to KITTI stereo inputs |
| Alternative geometry inputs | Offline ViPE poses/depth; online DynOSAM poses + completed KITTI depth |
| Dataset | Sequence | Annotated frames | |||
|---|---|---|---|---|---|
| FBMS-59 | camel01 | 6 | 91.28 | 99.64 | 95.46 |
| cars1 | 3 | 97.42 | 98.70 | 98.06 | |
| cars10 † | 4 | 69.29 | 63.50 | 66.40 | |
| cars4 † | 6 | 28.12 | 47.55 | 37.84 | |
| cars5 | 4 | 63.25 | 82.73 | 72.99 | |
| cats01 | 9 | 94.76 | 98.36 | 96.56 |
| Route | Geometry/depth | RAFT | DynOSAM | Residual |
|---|---|---|---|---|
| Controlled (GT pose + FS) | 180.71 | 27.81 | – | 0.48 |
| Offline (ViPE) | 529.46 | 27.81 | – | 0.48 |
| Online (DynOSAM) | Excluded | 27.81 | 69.40 | 0.48 |