Organizations: Beomseok Kang is with Seoul National University, Seoul, South Korea. · Niluthpol Chowdhury Mithun, Mikhail Sizintsev, Han-Pang Chiu, and Supun Samarasekera are with SRI International, Princeton, NJ, USA.
Fieldable low-light perception systems require both semantic and geometric scene understanding under severe domain shift and embedded compute constraints. We present FAMDA, a foundation-model-assisted self-training framework for unsupervised domain adaptation (UDA) in multi-task dense prediction. The framework integrates segmentation and depth foundation models into a self-training paradigm to generate high-quality pseudo-labels for the target domain, distilling their generalization capabilities into a single lightweight student network. To connect the method to deployment, we instantiate the adapted student on a compact handheld low-light prototype with DomiNite imaging and Jetson Orin Nano embedded compute. We evaluate on standard synthetic-to-real multi-task UDA benchmarks for comparison with prior work and on our field-collected low-light dataset to validate daytime-to-low-light adaptation in the target application setting. Across these evaluations, the proposed approach achieves strong segmentation and depth accuracy with substantially lower memory use and inference latency than direct foundation-model inference, demonstrating a practical path toward embedded multi-task perception for low-light field systems.
Figures & tables
Figure 1: Fieldable low-light multi-task perception. Top: our compact low-light sensing platform and onboard FAMDA pipeline, where a lightweight student model predicts semantic segmentation and monocular depth from a single low-light frame. Bottom: accuracy-efficiency tradeoff on SYNTHIA → Cityscapes, comparing model size against segmentation mIoU and depth RMSE with prior work. FAMDA provides strong performance with compact multi-task models.
Figure 2: Overview of FAMDA. FAMDA uses a self-training framework for multi-task UDA. A teacher network generates target-domain semantic pseudo-labels, which are refined using masks from a segmentation foundation model. A depth foundation model provides pseudo-depth supervision for target images. Both foundation models remain frozen. The resulting segmentation and depth supervision are distilled to a single lightweight student network , which alone is used at deployment.
Figure 3: Fieldable low-light multi-task perception pipeline. (a) Compact handheld prototype with DomiNite low-light imaging and embedded Jetson compute for onboard multi-task perception. (b) Onboard deployment pipeline: raw 16-bit low-light frames are preprocessed before being passed to the deployed FAMDA student model. At inference time, the student predicts semantic segmentation and monocular depth from a single frame.
Component
Configuration
Imaging
DomiNite low-light camera system ( 65∘ HFOV)
Camera configuration
Calibrated two-camera setup for data collection/evaluation
Network input
Single low-light frame
Predicted outputs
Semantic segmentation and monocular depth
Embedded compute
Jetson Orin Nano
Deployment format
PyTorch → ONNX → TensorRT
Table 1: Prototype and deployment configuration.
Method
Outputs
mIoU
RMSE
Model Size
A5000 Lat.
(%)
(m)
(MB)
(ms)
SSAM-H
Seg.
43.93
–
3351
34117
DAM-L
Depth
–
5.47
1279
60.8
DaF-Seg-B5
Seg.
52.92
–
325
35.5
DaF-Dep-B5
Depth
–
5.55
325
36.7
DaF-Seg/Dep-B5 †
Seg.+Depth
52.92
5.55
650
72.2
Table 2: Field-collected low-light dataset evaluation. FAMDA provides both semantic segmentation and monocular depth from one model. Metrics are mIoU (%, ↑ ) over 13 annotated classes and RMSE (m, ↓ ) over valid stereo-reference pixels. Model size and A5000 latency are reported for model-level comparison; embedded TensorRT runtime is reported in Tab. 3 .
Figure 4: Qualitative results on the field-collected low-light dataset. We compare our model with single-task foundation-model baselines: SSAM for semantic segmentation and DAM for depth estimation. Leveraging its joint outputs, FAMDA assigns the maximum depth to pixels segmented as sky. FAMDA produces both outputs from a single MiT-B5 model.
Model
TRT Size
Lat.
FPS
Peak Mem.
Power
mIoU
RMSE
(MB)
(ms)
(Hz)
(MB)
(W)
(%)
(m)
Ours-B0
23.4
123
8.1
1346
7.28
42.90
5.54
Ours-B1
42.5
144
6.9
1361
7.54
49.23
5.53
Ours-B2
66.0
158
6.3
1406
7.75
55.01
5.53
Ours-B5
187.0
212
4.7
1629
8.80
55.96
5.52
Table 3: Embedded TensorRT runtime and low-light metrics on Jetson Orin Nano. Runtime is measured on a Jetson Orin Nano 8GB using TensorRT FP16 at input resolution 526×840 . Latency is the standardized core runtime used for comparison across backbones, excluding file I/O, output saving, overlay blending, and display update. A representative Ours-B2 component-level timing breakdown, including raw-frame preprocessing and display-resolution colorization, is provided in the supplementary (Tab. S12). Peak memory reports RAM footprint above the idle baseline. Power denotes total board draw (VDD_IN) during sustained inference. mIoU and RMSE use the same field-collected low-light labels and stereo references as Tab. 2 .
Method
Backbone
SYN → CS (16 class)
VK2 → CS (8 class)
mIoU
RMSE
mIoU
RMSE
w/o UDA
STL target
DeepLabV2
67.93
6.62
77.10
6.62
STL source
DeepLabV2
35.63
13.79
58.77
11.99
MTL target
DeepLabV2
70.43
6.79
79.63
6.72
MTL source
DeepLabV2
15.32
14.51
49.50
12.26
w/ UDA
STL-UDA [ 35 ]
DeepLabV2
37.55
14.26
61.60
11.45
Table 4: Comparison on standard synthetic-to-real multi-task UDA benchmarks. The best and second-best UDA results are highlighted. Metrics are mIoU (%, ↑ ) and RMSE (m, ↓ ). Baseline results for STL, MTL, and prior UDA methods are cited from XTAM [ 24 ] and VTAGML [ 1 ] .
Semantic segmentation provides pixel-level scene understanding essential for autonomous driving and fine-grained perception tasks. However, training segmentation models requires costly, labor-intensive annotations on real-world datasets. Unsupervised Domain Adaptation (UDA) addresses this by training models on labeled synthetic data and adapting them to unlabeled real images. While conceptually simple, adaptation is challenging due to the domain gap, i.e., differences in visual appearance and scene structure between synthetic and real data. Prior approaches bridge this gap through pixel-level mixing or feature-level contrastive learning. Yet, these techniques suffer from two major limitations: (1) reliance on high-confidence pseudo-labels restricts learning to a subset of the target domain, and (2) prototype-based contrastive methods initialize class prototypes from source-trained models, yielding biased and unstable anchors during adaptation. To address these issues, we propose a dual-foundation UDA framework that leverages two complementary foundation models. First, we employ the Segment Anything Model (SAM) with superpixel-guided prompting to enable learning from a broader range of target pixels beyond high-confidence predictions. Second, we incorporate DINOv3 to construct stable, domain-invariant class prototypes through its robust representation learning. Our method achieves consistent improvements of +1.3% and +1.4% mIoU over strong UDA baselines on GTA-to-Cityscapes and SYNTHIA-to-Cityscapes, respectively.
Yerin Cheon, Aruna Balasubramanian, Francois Rameau
Stony Brook University, Stony Brook, NY, USA · SUNY Korea, The State University of New York, Korea
Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.
Jing Li, Pan Liu, Meng Zhao +7
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China · Engineering Research Center of Learning-Based Intelligent System, Ministry of Education of the People’s Republic of China, Tianjin University of Technology, Tianjin 300384, China · School of Artificial Intelligence, Tianjin University, Tianjin 300350, China +1
Low-visibility scenarios, such as low-light conditions, pose significant challenges to human pose estimation due to the scarcity of annotated low-light datasets and the loss of visual information under poor illumination. Recent domain adaptation techniques attempt to utilize well-lit labels by augmenting well-lit images to mimic low-light conditions. But handcrafted augmentations oversimplify noise patterns, while learning-based methods often fail to preserve high-frequency low-light characteristics, producing unrealistic images that lead pose models to generalize poorly to real low-light scenes. Moreover, recent pose estimators rely on image cues through image-to-keypoint cross-attention, but these cues become unreliable under low-light conditions. To address these issues, we propose Unsupervised Domain Adaptation for Pose Estimation (UDAPose), a novel framework that synthesizes low-light images and dynamically fuses visual cues with pose priors for improved pose estimation. Specifically, our synthesis method incorporates a Direct-Current-based High-Pass Filter (DHF) and a Low-light Characteristics Injection Module (LCIM) to inject high-frequency details from input low-light images, overcoming rigidity or the detail loss in existing approaches. Furthermore, we introduce a Dynamic Control of Attention (DCA) module that adaptively balances image cues with learned pose priors in the Transformer architecture. Experiments show that UDAPose outperforms state-of-the-art methods, with notable AP gains of 10.1 (56.4%) on the ExLPose-test hard set (LL-H) and 7.4 (31.4%) in cross-dataset validation on EHPT-XC. Code: https://github.com/Vision-and-Multimodal-Intelligence-Lab/UDAPose
Haopeng Chen, Yihao Ai, Kabeen Kim +3
University of Mississippi · National University of Singapore · Duksung Women’s University +1