Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models

Yu Jiang; Hanwen Jiang; Ahmed Abdelkader; Wen-Sheng Chu; Brandon Y. Feng; Zhangyang Wang; Qixing Huang

cs.CVApr 11, 2026

Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models

Authors: Yu Jiang, Hanwen Jiang, Ahmed Abdelkader, Wen-Sheng Chu, Brandon Y. Feng, Zhangyang Wang, Qixing Huang

Organizations: The University of Texas at Austin · Shanghai Jiao Tong University · Adobe Research · Google Research

Abstract

With the emergence of 3D foundation models, there is growing interest in fine-tuning them for downstream tasks, where LoRA is the dominant fine-tuning paradigm. As 3D datasets exhibit distinct variations in texture, geometry, camera motion, and lighting, there are interesting fundamental questions: 1) Are there LoRA subspaces associated with each type of variation? 2) Are these subspaces disentangled (i.e., orthogonal to each other)? 3) How do we compute them effectively? This paper provides answers to all these questions. We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation. We show that these subspaces are approximately disentangled. Integrating them leads to a reduced LoRA subspace that enables efficient LoRA fine-tuning with improved prediction accuracy for downstream tasks. In particular, we show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets. An ablation study validates the effectiveness of the choices in our approach.

Figures & tables

Figure 1: Our approach pre-computes LoRA subspaces associated with each type of 3D dataset variation in geometry, texture, camera, and lighting via curated synthetic datasets. These subspaces are integrated into a reduced LoRA basis for efficient fine-tuning.

Figure 2: Singular values of

C

computed from QKV and projection matrices of the 17-th global self-attention layer of 10 LoRAs with respect to geometry variations. Note that the singular values are reported in log-scale.

Panel of Figure 3 — Figure 3: Examples of synthetic datasets for each type of variations. We empirically push the variation in each type to the extreme while maintaining small variations among other types. Note that these images are very different from real-world images.

Figure 4: There are four spectral patterns of matrix

C

across some layers with respect to different dataset variations. They are colored in red, green, blue, and black. ’tex-g-atten-qkv-1’ indicates the QKV matrix in the first global attention layer fine-tuned on texture variations. Note that we have applied a log scale to the singular values. It is clear that there is a significant drop in these curves, which indicates the existence of shared subspaces.

Figure 5: Maximum singular values of each shared LoRA subspace. Linear layers, QKV, and projection matrices of global self-attention and local self-attention are plotted in the top and bottom rows, respectively. We show four curves in each plot, which correspond to texture, geometry, camera, lighting variations.

Figure 6: We show overlap ratios between six pairs of four subspaces that correspond to variations in geometry, texture, camera, and lighting. We show three representative layers of the global QKV attention. This metric, akin to reprojection error, reveals that each pair of subspaces is approximately orthogonal. Notably, the orthogonality is most significant between texture and camera, as well as between geometry and camera.

Method	# Trainable Param.	Synthetic Face Dataset			Real Face Dataset
Method	# Trainable Param.	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
VGGT	-	9.006	4.965	80.74	2.651	98.59
Full	853.6 M	5.585	3.531	85.77	2.203	98.85
LoRA (rank=16)	16.3 M	5.767	3.385	84.78	2.115	98.92
LoRA (rank=32)	32.7 M	6.251	3.841	84.64	2.159	98.93
LoRA (rank=64)	65.3 M	6.393	3.971	84.59	2.157	98.94

Table 1: Evaluation on Synthetic and Real-World Human Face Datasets under Micro-Baseline Settings. The synthetic dataset contains 50 face images, which are used to render videos from nine different viewpoints. The real-world dataset consists of 50 printed images, each captured in one long-burst bundle. Bold : best; underline : second best.

Figure 7: Qualitative Comparison of 2D Face Anti-Spoofing Tasks. Compared to other fine-tune strategies, our method produces more accurate and robust reconstruction with fewer artifacts.

Figure 8: Visual Results of Clothed Human Reconstruction Tasks. Each input consists of eight different viewpoints. The first row is selected from the THuman 2.1 test split, while the second row is chosen from the 2K2K test set. Our model produces fewer artifacts on the object, but its performance on more detailed regions, such as the hands, is less ideal.

Method	# Trainable Param.	THuman (In-domain)			2K2K (Cross-domain)
Method	# Trainable Param.	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
VGGT	-	2.816	1.911	91.51	3.103	2.122	92.81
Full	853.6 M	3.053	1.932	91.17	3.655	2.213	92.25
LoRA (rank=16)	16.3 M	3.195	2.089	91.63	2.717	1.829	92.73
LoRA (rank=32)	32.7 M	2.849	1.922	91.85	2.633	1.773	93.00
LoRA (rank=64)	65.3 M	2.791	1.902	92.12	3.017	1.968	93.18

Table 2: Evaluation of Clothed Human Reconstruction: Point Map Estimation on THuman 2.1 [ 32 ] and 2K2K [ 12 ] . Our method achieves the best performance across nearly all metrics.

Method	# Trainable Param.	In-domain
Method	# Trainable Param.	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
VGGT	-	3.123	3.271	67.77
Fully fine-tuned	853.6 M	1.653	2.559	74.18
LoRA (rank=16)	16.3 M	1.808	2.522	73.74
LoRA (rank=32)	32.7 M	1.811	2.562	73.76
LoRA (rank=64)	65.3 M	1.787	2.580	73.81

Table 3: Evaluation of Transparent Object Reconstruction: Point Map Estimation on ClearPose [ 8 ] . Our method achieves comparable performance across all metrics.

Method	THuman (In-domain)			2K2K (Cross-domain)
Method	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
PSV ( $d$ =32)	5.605	3.086	89.58	6.844	4.523	89.27
PSV ( $d$ =64)	4.066	2.423	90.65	3.805	2.202	90.84
PSV ( $d$ =128)	3.709	2.394	91.10	3.290	2.128	92.50
PSV ( $d$ =256)	2.785	1.904	91.76	2.542	1.762	93.42
Ours ( $d$ =8)	5.839	3.363	89.33	5.712	2.602	89.98

Table 4: Computing

\overline{A}

and

\overline{B}

using our approach is superior to using the leading singular vectors of the original weight matrix.

Method	THuman (In-domain)			2K2K (Cross-domain)
Method	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
r=64 ( $d$ =8)	6.621	3.092	88.76	6.070	3.299	88.94
r=64 ( $d$ =16)	4.030	2.343	89.94	4.813	3.031	89.93
r=64 ( $d$ =32)	3.797	2.482	91.31	3.246	2.094	92.43
r=64 ( $d$ =64)	2.783	1.871	91.41	2.563	1.756	93.19
r=16 ( $d$ =8)	5.839	3.363	89.33	5.712	2.602	89.98

Table 5: Comparison of Subspaces Extracted from LoRAs with Different Ranks. Ablation study to validate our shared space hypothesis.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: More visual comparison of 2D Face Anti-Spoofing Tasks.

Method	THuman (In-domain)			2K2K (Cross-domain)
Method	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
Uniform ( $d$ =16)	3.392	2.138	90.91	3.019	1.887	91.70
Uniform ( $d$ =32)	3.332	2.220	91.56	2.825	1.878	93.00
Uniform ( $d$ =64)	2.745	1.882	91.82	2.513	1.754	93.56
Importance ( $d$ =16)	4.088	3.783	88.81	3.822	5.900	90.29
Importance ( $d$ =32)	3.778	2.444	91.23	3.165	2.041	92.41

Appendix

Table 1: Comparison Between Different Rank Allocation Strategies. The overall trend is the same, with minimal performance differences.

Figure 2: More visual comparison of Clothed Human Reconstruction Tasks.

Figure 3: Singular values of the QKV matrix in the global attention layer with respect to texture variations.

Figure 4: Singular values of the QKV matrix in the frame attention layer with respect to texture variations.

Figure 5: Singular values of the projection matrix in the global attention layer with respect to texture variations.

Figure 6: Singular values of the projection matrix in the frame attention layer with respect to texture variations.

Figure 7: Singular values of the first fully connected matrix in the global attention layer with respect to texture variations.

Figure 8: Singular values of the first fully connected matrix in the frame attention layer with respect to texture variations.

Figure 9: Singular values of the second fully connected matrix in the global attention layer with respect to texture variations.

Figure 10: Singular values of the second fully connected matrix in the frame attention layer with respect to texture variations.

Figure 11: Singular values of the QKV matrix in the global attention layer with respect to geometry variations.

Figure 12: Singular values of the QKV matrix in the frame attention layer with respect to geometry variations.

Figure 13: Singular values of the projection matrix in the global attention layer with respect to geometry variations.

Figure 14: Singular values of the projection matrix in the frame attention layer with respect to geometry variations.

Figure 15: Singular values of the first fully connected matrix in the global attention layer with respect to geometry variations.

Figure 16: Singular values of the first fully connected matrix in the frame attention layer with respect to geometry variations.

Figure 17: Singular values of the second fully connected matrix in the global attention layer with respect to geometry variations.

Figure 18: Singular values of the second fully connected matrix in the frame attention layer with respect to geometry variations.

Figure 19: Singular values of the QKV matrix in the global attention layer with respect to camera variations.

Figure 20: Singular values of the QKV matrix in the frame attention layer with respect to camera variations.

Figure 21: Singular values of the projection matrix in the global attention layer with respect to camera variations.

Figure 22: Singular values of the projection matrix in the frame attention layer with respect to camera variations.

Figure 23: Singular values of the first fully connected matrix in the global attention layer with respect to camera variations.

Figure 24: Singular values of the first fully connected matrix in the frame attention layer with respect to camera variations.

Figure 25: Singular values of the second fully connected matrix in the global attention layer with respect to camera variations.

Figure 26: Singular values of the second fully connected matrix in the frame attention layer with respect to camera variations.

Figure 27: Singular values of the QKV matrix in the global attention layer with respect to lighting variations.

Figure 28: Singular values of the QKV matrix in the frame attention layer with respect to lighting variations.

Figure 29: Singular values of the projection matrix in the global attention layer with respect to lighting variations.

Figure 30: Singular values of the projection matrix in the frame attention layer with respect to lighting variations.

Figure 31: Singular values of the first fully connected matrix in the global attention layer with respect to lighting variations.

Figure 32: Singular values of the first fully connected matrix in the frame attention layer with respect to lighting variations.

Figure 33: Singular values of the second fully connected matrix in the global attention layer with respect to lighting variations.

Figure 34: Singular values of the second fully connected matrix in the frame attention layer with respect to lighting variations.

Figure 35: The overlap ratio between subspaces that correspond to variations in geometry and texture.

Figure 36: The overlap ratio between subspaces that correspond to variations in geometry and camera motion.

Figure 37: The overlap ratio between subspaces that correspond to variations in geometry and lighting.

Figure 38: The overlap ratio between subspaces that correspond to variations in texture and camera motion.

Figure 39: The overlap ratio between subspaces that correspond to variations in texture and lighting.

Figure 40: The overlap ratio between subspaces that correspond to variations in camera motion and lighting.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 605 papers

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
Jul 15, 2026Haobo Zhang, Jiankun Wang, Suraj Rajendran +5Subspace Optimization Federated Learning Aggregation
University of Michigan · Cornell University · University of Pennsylvania
LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition
Oct 20, 2026Thamirawaran Sathiyalogeswaran, Navindu De Silva, Chamikara Siriwardane +4Low-Rank Adaptation LLM Inference Acceleration
University of Moratuwa, Sri Lanka · University of Michigan, USA
Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning
Jun 16, 2026Sneha Paul, Zachary Patterson, Nizar BouguilaMultimodal Robustness VLM Adaptation
Concordia University, Canada

Jul 15, 2026J/K move · Enter open · S save

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

Haobo Zhang, Jiankun Wang, Suraj Rajendran, Weishen Pan+4

University of Michigan · Cornell University · University of Pennsylvania

Subspace Optimization Federated Learning Aggregation Low-Rank Adaptation Federated Fine-Tuning Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients' insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning