Parameter-Efficient Temporal Domain Generalization

Zekun Cai; Xinyuan Song; Yiheng Yao; Xuan Song; Ryosuke Shibasaki; Liang Zhao

cs.LGFeb 12, 2026

Parameter-Efficient Temporal Domain Generalization

Authors: Zekun Cai, Xinyuan Song, Yiheng Yao, Xuan Song, Ryosuke Shibasaki, Liang Zhao

Organizations: The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan · Emory University, Atlanta, GA, USA · Jilin University, Changchun, China

Abstract

Temporal domain generalization (TDG) learns from a sequence of observed domains to predict models for unseen future domains. Existing parameter-dynamics methods become prohibitively expensive at modern backbone scales because their temporal encoders and generators operate on the entire adapted parameter state. Replacing full-model updates with LoRA does not remove this failure mode: the dynamics model must still encode and reconstruct every adapter parameter, making the auxiliary model itself prohibitively large for modern backbones. We propose Low-rank Time-varying Core (LoTiC), a parameter-efficient TDG method that decomposes temporal updates into shared low-rank bases and a compact core. The bases define a stable update subspace, while only the core evolves. LoTiC therefore learns temporal dynamics in an r*r space instead of tracking the full adapter. We justify this factorization through the geometry of parameter increments and support continuous, recurrent, and time-conditioned core models. Experiments span six controlled and real-world benchmarks, three data modalities, classification and generation, and backbones from 66M to 8B parameters. LoTiC ranks first in 14 of 15 settings, including near-perfect future-domain accuracy with an 8B backbone on Rotating 2-Moons. It adds only 0.13--0.51% trainable parameters beyond LoRA and trains 1.8--7.1 $\times$ faster than parameter-dynamics baselines. These results establish that temporal parameter dynamics can remain both learnable and efficient as pretrained backbones scale.

Figures & tables

Figure 1: (a) Independent domain-wise LoRA optimization can yield scattered updates without temporal consistency. (b) With shared bases $B$ and $A$ , temporal adaptation is captured by the compact core $F_{t}$ . The temporal core can be instantiated by different sequence modeling modules.

Figure 2: Controlled temporal extrapolation on Rotating 2-Moons. (a) Twenty domains with 15 observed and 5 unseen. (b) Future-domain decision boundaries of Koodos-LoRA and LoTiC. (c) Domain-wise accuracy; the observed and unseen intervals use different x-axis scales.

Dataset	Backbone Scale	ERM	Last Domain	IncFT	DRAIN-L	Koodos-L	FreKoo-L	Prompt TDG	TEA	LoTiC
Rot-2- Moons	DistilBERT 66M	18.5 (1.0)	12.7 (0.9)	28.2 (6.5)	21.4 (5.9)	11.9 (0.5)	11.8 (1.1)	25.9 (4.2)	21.8 (3.5)	0.6 (0.3)
	Qwen3 0.6B	23.5 (4.4)	13.7 (1.3)	24.7 (1.0)	OOM	OOM	OOM	49.8 (15.9)	25.6 (1.3)	2.4 (2.5)
	LLaMA-3 8B	21.8 (6.1)	12.7 (0.6)	22.1 (7.2)	OOM	OOM	OOM	23.4 (5.4)	17.3 (1.2)	0.9 (0.7)
Rot- MNIST	DINOv2-B 86M	20.0 (1.2)	45.9 (2.0)	44.7 (2.0)	61.8 (2.5)	61.3 (5.2)	28.7 (1.0)	19.1 (1.4)	18.7 (0.6)	8.6 (2.4)
	DINOv2-L 0.3B	16.6 (1.9)	37.5 (3.3)	37.8 (2.7)	OOM	OOM	OOM	15.5 (1.1)	15.6 (0.4)	7.3 (1.0)

Table 1: Future-domain performance, reported as mean (standard deviation). Classification benchmarks use error rate (%,

\downarrow

); WMT uses ROUGE-L F1 (

\uparrow

). Bold and underlined values are best and second-best. -L denotes LoRA; OOM denotes out of memory.

Method	Total params	Trainable params	Trainable vs. LoRA	Ckpt. (MB)	Train (s)	Test (s)	Test err. (%)
ERM	66.59M	225,026	1.00 $\times$	0.900	10	$<0.1$	18.5
DRAIN–LoRA	182.01M	115.64M	513.91 $\times$	462.6	165.3	$<0.1$	21.4
Koodos–LoRA	532.0M	465.65M	2069.30 $\times$	1862.6	580	$<0.1$	11.9
FreKoo–LoRA	763.0M	696.67M	3095.95 $\times$	2786.7	150	$<0.1$	11.8
LoTiC	66.59M	226,178	1.01 $\times$	0.905	82	$<0.1$	0.6

Table 2: Efficiency on Rot-2-Moons (DistilBERT, 66M).

Backbone	LoRA	LoTiC	Additional
DistilBERT (66M)	225,026	226,178	+1,152 (0.51%)
DINOv2-B (86M)	450,058	452,362	+2,304 (0.51%)
DINOv2-L (0.3B)	1,189,898	1,194,506	+4,608 (0.39%)
Qwen3 (0.6B)	1,610,754	1,616,130	+5,376 (0.33%)
LLaMA-3 (8B)	4,739,074	4,745,218	+6,144 (0.13%)

Table 3: Parameter overhead by backbone.

Figure 3: Efficiency at scale on a controlled LLaMA-3 family. Model scale is varied only by increasing the number of identical Transformer blocks. (a) Peak GPU memory. (b) Training time relative to the smallest model. Crosses indicate OOM on a 48-GB GPU.

Figure 4: Sensitivity and convergence on Rotating 2-Moons with DistilBERT. Error bars and shading show the standard deviation over three seeds; the dashed line is the strongest baseline.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Backbone	Targets	$r$	$\alpha$	Precision	LR	Epochs	Batch
Rotating 2-Moons	DistilBERT	$q/k/v$	8	64	FP32	$10^{-3}$	200	200
	Qwen3–0.6B	$q/k/v$	8	64	FP32	$10^{-3}$	500	200
	LLaMA-3–8B	$q/k/v$	8	128	BF16/FP32	$10^{-3}$	500	200
Rotated MNIST	DINOv2-B/L	$q/k/v$	8	64	FP32	$10^{-3}$	300	200
Yearbook	DINOv2-B/L	$q/k/v/o+\mathrm{FFN}$	8	32	FP32	$3\!\times\!10^{-4}$	100	128
HuffPost	DistilBERT	$q/k/v/o+\mathrm{FFN}$	8	64	FP32	$5\!\times\!10^{-4}$	10	128

Appendix

Table 5: Dataset-specific configurations. BF16/FP32 denotes BF16 backbone computation with FP32 trainable modules.

Method	Evolved state	Auxiliary parameters	Training trajectory
DRAIN–LoRA	$p$	$\mathcal{O}(ph+h^{2})$	$\mathcal{O}(Tp)$
Koodos–LoRA	$p$	$\mathcal{O}(ph+z^{2})$	$\mathcal{O}(Tp)$
FreKoo–LoRA	$p$	$\mathcal{O}(ph+z^{2})$	$\mathcal{O}(Tp)$
LoTiC (Flow)	$r^{2}$ per matrix	$Mr^{2}$	$\mathcal{O}(p+Mr^{2})$

Appendix

Table 6: Asymptotic cost of temporal modeling.

Domain Generalization

Domain generalization develops models that perform reliably in new environments without training on target-domain data.

See all 375 papers

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
Oct 13, 2026Yangming Zhang, Ling Yi, Datong Li +4Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China
EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning
Aug 2, 2026Srinivas Anumasa, Dianbo LiuUncertainty Quantification Low-Rank Adaptation
National University of Singapore
Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
May 15, 2026Jing Gao, Zhong-Yi Lu, Pan Zhang +1LLM Fine-Tuning Low-Rank Adaptation
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China · School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China +1

Oct 13, 2026J/K move · Enter open · S save

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Yangming Zhang, Ling Yi, Datong Li, Jinliang Ding+3

Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA variants reveals that these residual components account for most of the gradient energy and exhibit strong spectral concentration. Motivated by this finding, we propose Dynamic Tangent LoRA (DT-LoRA), which partitions a fixed rank budget into a stable branch and a dynamic branch to capture the principal residual gradient directions. We show that these directions define the optimal update of a local quadratic model under orthogonal-complement and rank constraints. Matrix-free randomized SVD recovers them without materializing the residual gradient, while controlled write-back preserves accumulated updates and stabilizes training within the allocated rank. DT-LoRA adds no trainable parameters or inference cost over LoRA. Experiments across computer vision and natural language processing tasks demonstrate consistent improvements over LoRA and its variants. Moreover, despite being a general-purpose adaptation method, DT-LoRA outperforms task-specific approaches to open-vocabulary domain generalization for semantic segmentation, achieving state-of-the-art results. Code is available at https://github.com/Snipine/DT-LoRA.

Abstract

Domain Generalization

Explore similar work

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

Strategic Over-Parameterization for Generalizable Low-Rank Adaptation