Axis-LoRA: Probing and Adapting Embedding Models for Personal-Memory Retrieval
Organizations: Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Mobile eXperience Business, Samsung Electronics · ASRI/INMC/AIIS, Seoul National University
Abstract
As LLMs are increasingly asked questions about a user's personal experiences, retrieval must distinguish memories by time, place, person, and event. Current personal-memory pipelines rely on frozen dense embedding models to embed memories for retrieval, but we find that these models do not capture the four axes well. Swapping a memory's date or place barely shifts its embedding, and a trained projection on top of the frozen embedding does not recover the missing signal. The problem lives inside the model, where each axis is encoded by a distinct attention-head pattern. Guided by these findings, we propose Axis-LoRA, a per-axis adapter that places four LoRA branches inside the embedding model and combines them via a per-query gate. Across three standard embedding models evaluated on LongMemEval, ATM-Bench, and LoCoMo, Axis-LoRA improves 4-benchmark mean R@10 by to ~pp while training only a small fraction of the base parameters on a small axis-counterfactual training set. Code is available at https://github.com/dohoonkim-ai/axis-lora.
Figures & tables
| I met Marco at the Boston conference on 2024-06-15 to discuss the rehearsal dinner. | |
|---|---|
| “2024-06-15” “2024-09-20” | |
| “Boston” “Seattle” | |
| “Marco” “Sarah” | |
| “rehearsal dinner” “budget meeting” |
| Encoder | TEMP. | SPAT. | ENT. | EVENT |
|---|---|---|---|---|
| Multiple of non-axis 1-word edit floor | ||||
| MiniLM ( ) | ||||
| bge-base ( ) | ||||
| Qwen3-Emb-0.6B ( ) | ||||
| Percentage of inter-memory distance | ||||
| MiniLM ( ) | ||||
| MiniLM | bge-base | Qwen3-0.6B | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Axis | raw | proj | raw | proj | raw | proj | |||
| TEMP. | |||||||||
| SPAT. | |||||||||
| ENT. | |||||||||
| EVENT | |||||||||
| Model | TEMP. | EVENT | SPAT. | ENT. |
|---|---|---|---|---|
| MiniLM | 41.7% | 36.1% | 29.2% | 26.4% |
| bge-base | 39.6% | 35.4% | 29.2% | 30.6% |
| Qwen3-0.6B | 32.1% | 25.4% | 19.6% | 18.5% |
| MiniLM-L6-v2 | bge-base-en-v1.5 | Qwen3-Embedding-0.6B | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Bench | Base | Axis-LoRA | Base | Axis-LoRA | Base | Axis-LoRA | |||
| LME | |||||||||
| ATM-N | |||||||||
| ATM-H | |||||||||
| LoCoMo | |||||||||
| Avg. | |||||||||
| (pp) | CI | |
|---|---|---|
| 4-benchmark mean | ||
| MiniLM-L6-v2 | ||
| bge-base-en-v1.5 | ||
| Qwen3-0.6B | ||
| ATM-Hard, 3-encoder pooled | ||
| Setup | NFC. | SciF. | ArgA. | SciD. | FiQA | TREC-C | Avg. |
|---|---|---|---|---|---|---|---|
| Base | |||||||
| Axis-LoRA |
| Method | LME | ATM-N | ATM-H | LoCoMo | Avg. |
|---|---|---|---|---|---|
| Base | |||||
| Axis-prefix prompt | |||||
| TF-PCA | |||||
| MLP+InfoNCE | |||||
| Shared-trunk LoRA | |||||
| Axis-LoRA |
| Setting | LME | ATM-N | ATM-H | LoCoMo | Avg. |
|---|---|---|---|---|---|
| Base | |||||
| Axis-LoRA | |||||
| axis-CF supervision | |||||
| KD anchor | |||||
| gate (uniform routing) |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Ratio | Raw | Jaccard-normalized |
|---|---|---|
| ENTITY / TEMPORAL | ||
| SPATIAL / TEMPORAL | ||
| EVENT / TEMPORAL |
| Backbone | E / T ratio range across variants |
|---|---|
| MiniLM | – |
| Qwen3-0.6B | – |
| Qwen3-8B | – |
| MiniLM | bge-base | Qwen3-0.6B | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Axis | raw | proj | raw | proj | raw | proj | |||
| TEMP. | |||||||||
| SPAT. | |||||||||
| ENT. | |||||||||
| EVENT | |||||||||
| Model | Full pool E / T % | Mid-position pool E / T % |
|---|---|---|
| MiniLM | (ratio ) | (ratio ) |
| Qwen3-0.6B | (ratio ) | (ratio ) |
| Gate | LME | ATM-N | ATM-H | LoCoMo | Avg. |
|---|---|---|---|---|---|
| Sparsemax (Ours) | |||||
| Softmax + axis-CE |
| MiniLM | bge-base | Qwen3-0.6B | |
|---|---|---|---|
| Single adapter (control) | |||
| Axis-LoRA (paper) | |||
| + axis-sup. gate | |||
| same, uniform routing | |||
| Qwen3-0.6B routing decomposition | |||
| free gate: learned / uniform | / | ||
| LoCoMo ( temporal , ) | LME ( temp.-reasoning , ) | |||
|---|---|---|---|---|
| time | other | time | other | |
| MiniLM | ||||
| bge-base | ||||
| Qwen3-0.6B | ||||
| MiniLM | bge-base | Qwen3-0.6B | |||||
|---|---|---|---|---|---|---|---|
| Axis | Setup | R@10 | med | R@10 | med | R@10 | med |
| TEMP | raw | ||||||
| MLP | |||||||
| A-LoRA | |||||||
| SPAT | raw | ||||||
| MLP | |||||||
| Variant | LME | ATM-N | ATM-H | LoCoMo | Avg. |
|---|---|---|---|---|---|
| Axis-LoRA | |||||
| Shared-trunk | |||||
| MLP+InfoNCE | |||||
| axis-CF supv. | |||||
| KD anchor |
| MiniLM | bge-base | Qwen3-0.6B | |
|---|---|---|---|
| Temporal-only | |||
| Temporal-only (step-matched) | |||
| Joint 4-axis Axis-LoRA |
| Shared trunk, | ||||||
|---|---|---|---|---|---|---|
| Shared trunk, |
| Benchmark | Split | Queries | Pool | Avg evid. | Multi-evid. |
|---|---|---|---|---|---|
| LongMemEval | — | ||||
| LoCoMo | — | ||||
| ATM-Bench | Normal | ||||
| ATM-Bench | Hard |
| MiniLM-L6-v2 | bge-base-en-v1.5 | Qwen3-Embedding-0.6B | |
|---|---|---|---|
| Max sequence length | |||
| Per-GPU batch size | |||
| Gradient accumulation | |||
| Effective batch ( GPU) | |||
| Epochs | |||
| Learning rate (gate) |
| Setup | NFC. | SciF. | ArgA. | SciD. | FiQA | TREC-C | Avg. | |
| MiniLM-L6-v2 | ||||||||
| Base | — | |||||||
| Axis-LoRA | ||||||||
| bge-base-en-v1.5 | ||||||||
| Base | — | |||||||
| Axis-LoRA | ||||||||
| R@1 | R@5 | R@10 | R@100 | MRR | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Encoder | Bench | Base | Ours | Base | Ours | Base | Ours | Base | Ours | Base | Ours | |||||
| MiniLM | LME | |||||||||||||||
| ATM-N | ||||||||||||||||
| ATM-H | ||||||||||||||||
| LoCoMo | ||||||||||||||||
| bge-base | LME | |||||||||||||||
| ATM-Hard ( ) | 3-bench mean (excl. ATM-H) | 4-bench mean | ||||
|---|---|---|---|---|---|---|
| 95% CI | 95% CI | 95% CI | ||||
| MiniLM | ||||||
| bge-base | ||||||
| Qwen3-0.6B | ||||||
| 3-encoder pooled | ||||||
| Cell | Unchanged | Wins / losses | Winner rank | Loser rank |
|---|---|---|---|---|
| Qwen3 ATM-N | ( ) | / | ||
| bge LoCoMo | ( ) | / |
| temporal-reasoning | temporal | |
|---|---|---|
| (LME, ) | (LoCoMo, ) | |
| MiniLM | ||
| bge-base | ||
| Qwen3-0.6B |
| TEMPORAL | SPATIAL | ENTITY | EVENT | |
|---|---|---|---|---|
| MiniLM | / | / | / | / |
| bge-base | / | / | / | / |
| Qwen3-0.6B | / | / | / | / |
| LME (judge acc.) | ATM-N (QS) | LoCoMo (F1) | |
|---|---|---|---|
| MiniLM | |||
| bge-base | |||
| Qwen3-0.6B |
Text Embedding Models
Text embedding models map words, sentences, or documents to vectors for semantic search, retrieval, clustering, and downstream prediction.
See all 136 papers