HARP-LoRA: Hierarchy-Conditioned Rank Projection for Low-Rank Adaptation in Scientific Table Understanding
Authors: Hossein Sholehrasa, Mohammadhossein Amini, Doina Caragea, Remya Ampadi Ramachandran, Dan Wu, Zihang Shi, Zelin Li, Lindsey Hulbert, Eder Martinez, Xuan Xu, Majid Jaberi-Douraki
Organizations: 1DATA Consortium and FARAD Program, Kansas State University, Olathe, KS, USA · Department of Computer Science, Kansas State University, Manhattan, KS, USA · Olin Business School, Washington University in St. Louis, St. Louis, Missouri, USA · Department of Mathematics, Kansas State University, Manhattan, KS, USA · Institute for Health Informatics, University of Minnesota Medical School, Minneapolis, MN, USA · Miami Herbert Business School, University of Miami, Coral Gables, Florida, USA · Department of Animal Sciences and Industry, Kansas State University, Manhattan, KS, USA · Department of Statistics, Kansas State University, Manhattan, KS, USA
Tabular data is widely used in scientific applications, where complex layouts, including multi-level headers, merged cells, captions, and footnotes, make information extraction difficult. This is particularly important in pharmacokinetics (PK), where values must be correctly associated with the drug, species, dose, administration route, unit, and experimental condition. Parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) reduce adaptation cost but do not explicitly incorporate relationships between table values and surrounding headers. We introduce HARP-LoRA, a hierarchy-conditioned rank projection method for low-rank adaptation in scientific table understanding. HARP-LoRA extends projected LoRA by constructing token-specific intrinsic representations from the row- and column-header hierarchies of each data token. These representations condition the low-rank down-projection, incorporating hierarchical context while keeping the pretrained model and random projection tensors frozen. We evaluate HARP-LoRA on 950 manually annotated PK tables from 587 scientific articles and on WikiTableQuestions (WikiTQ) using Llama3.2 3B, Llama3.1 8B, and Gemma3 27B. With Llama3.1 8B, HARP-LoRA achieves a PK F1-score of 57.64% and WikiTQ accuracy of 53.15%, improving over LoRA by 1.42 and 6.56 points, respectively. With Gemma~3 27B, it achieves the best PK F1-score of 70.63% and WikiTQ accuracy of 56.87%. These results use only 172K, 197K, and 547K trainable parameters, far fewer than LoRA and TableLoRA. Overall, conditioning lightweight adapters on table-header hierarchies improves extraction from scientific tables and reasoning over flatter general-domain tables while maintaining parameter efficiency. Code is available at: https://github.com/hosseinsholehrasa/HARP-LoRA
Figures & tables
Fig. 1: Overview of HARP-LoRA. For each table data token, the corresponding hierarchical row and column headers are identified and encoded separately. Their representations are combined using separate trainable vectors to produce a token-specific intrinsic vector, which conditions the projected LoRA down-projection. The resulting low-rank representation is mapped back to the model hidden space using a shared up-projection.
Model
Method
PK Dataset
WikiTQ
# Train. Params.
P (%)
R (%)
F1 (%)
Acc (%)
Llama3.2 3B
LoRA
53.57
58.76
53.57
44.17
1.84M (0.057%)
TableLoRA 2D
53.13
57.90
52.87
43.80
2.59M (0.080%)
projected LoRA
52.26
57.34
52.20
47.32
115K (0.004%)
HARP-LoRA
53.26
58.09
53.08
45.23
172K (0.005%)
Llama3.1 8B
LoRA
57.12
59.98
56.22
46.59
2.62M (0.033%)
TABLE I: Performance comparison on the PK and WikiTQ datasets across three backbone models. HARP-LoRA achieves the strongest results for the 8B and 27B backbones while using substantially fewer trainable parameters than LoRA and TableLoRA 2D. Best results within each backbone are shown in bold , and second-best results are underlined . P, R, F1, and Acc denote precision, recall, F1-score, and answer accuracy, respectively.
Intrinsic Dimension k
F1 (%)
256
30.68
512
55.23
1024
57.64
2048
56.79
TABLE II: Effect of the intrinsic dimension k on the PK data F1-score for HARP-LoRA with Llama 3.1 8B. Performance improves substantially as k increases from 256 to 1024, with the best F1-score achieved at k=1024.
Configuration
F1 Score (%)
Full model
57.64
Without row-header hierarchy
56.91
Without column-header hierarchy
56.36
Without level-preserving pooling
56.58
Without RMS normalization
56.97
Shared intrinsic vector (projected LoRA)
56.78
TABLE III: Ablation results on the PK dataset using Llama 3.1 8B. Removing any component reduces the F1-score, with the largest degradation observed when the column-header hierarchy is excluded. The full HARP-LoRA model achieves the best performance.
Large language models (LLMs) have demonstrated strong performance in table understanding. However, they typically process table content and headers as linearized token sequences. This representation weakens the two-dimensional and hierarchical structural relationships encoded by multi-level row and column headers. Existing parameter-efficient fine-tuning methods incorporate basic row and column information but do not explicitly capture the rich structural dependencies induced by hierarchical table headers. We propose DeepTable, a structure-aware approach for table understanding with LLMs. DeepTable comprises two complementary components. Structural Attention Bias (SAB) introduces learnable biases into the attention logits to explicitly represent whether pairs of table tokens share the same row or column. Tree Path Encoding (TPE) represents each table token using the ancestor paths of its row and column headers, preserving its position within the multi-level table structure. We integrate DeepTable with TableLoRA (He et al., 2025) to inject structural information into parameter-efficient adaptation. Across three LLM backbones, DeepTable consistently improves the corresponding TableLoRA baselines on three table question answering benchmarks, achieving average gains of 7.42 points on HiTab, 3.23 points on WikiTQ, and 2.01 BLEU points on FeTaQA. These results demonstrate the effectiveness of the proposed structural biases across different LLM backbones.
Jyun-Ying Yen, Cheng-Kuan Lin, Yu-Chee Tseng
Department of Computer Science National Yang Ming Chiao Tung University, Taiwan
Tables are ubiquitous across diverse domains, yet reasoning over them remains a significant challenge for modern large language models (LLMs). Current approaches typically linearize tables into sequences, inherently overlooking their intrinsic two-dimensional and hierarchical structure. To address this, we propose H2Table (Hierarchical Hypergraph-Enhanced Table Reasoning), a novel framework that represents complex tables as hierarchical nested hypergraphs. To process this representation, we design a tailored hypergraph encoder to facilitate message passing between hyperedges (headers) and nodes (cells), thereby perceiving the semantic entailment relationships between them within complex tables. Furthermore, we introduce a set of learnable query vectors acting as a lightweight bridge to extract representative structural embeddings from the encoder into the LLM. Experimental results demonstrate that our approach effectively handles complex table question answering tasks with hierarchical nested headers. Notably, on the HiTab dataset, H2Table achieves an average improvement of 22.88% over state-of-the-art baselines on highly complex tables with a nesting depth of four. Our code is available at: https://github.com/lila120/h2table.
Jia Ling, Yangfan Wang, Chen Tang +4
Harbin Institute of Technology · AI Research Center, Midea Group (Shanghai) Co., Ltd. · Changchun University of Science and Technology +1
With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank r requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient. In this paper, we propose MatryoshkaLoRA, a general, Matryoshka-inspired training framework for LoRA that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix P between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing P and ensures all sub-ranks embed the available gradient information efficiently. Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy-performance trade-offs across ranks on the evaluated datasets. Our code is available at https://github.com/IST-DASLab/MatryoshkaLoRA.
Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh
Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA