LiFT: How to Enable In-Context Learning for Longitudinal Modelling

Iqra Ali; Talia Tseriotou; Mahmud Elahi Akhter; Yuxiang Zhou; Maria Liakata

cs.CLMar 25, 2026

LiFT: How to Enable In-Context Learning for Longitudinal Modelling

Authors: Iqra Ali, Talia Tseriotou, Mahmud Elahi Akhter, Yuxiang Zhou, Maria Liakata

Organizations: Queen Mary University of London (UK) · The Alan Turing Institute (UK)

Abstract

Longitudinal NLP tasks such as mental health monitoring and stance evolution require modeling temporally ordered text to track persistence and detect change. Such tasks also suffer from data scarcity, often involving rare events and sparsely annotated data. Large Language Models (LLMs) can learn from small amounts of data through in-context learning (ICL), which is particularly important in low-data resource scenarios. However, when it comes to tracking evolving interactions or identifying rare events or changes, LLMs fall short in both zero- and few-shot ICL. To address this limitation, we introduce LiFT, a model-agnostic longitudinal Instruction Fine-Tuning framework that is able to induce ICL behaviour in LLMs for longitudinal tasks. LiFT unifies diverse tasks through a shared instruction schema that leverages sequential dependencies, curriculum learning, and temporal conditioning. We evaluate LiFT on five longitudinal datasets under zero-, one-, and three-shot settings, demonstrating that LiFT delivers superior performance to IFT models under identical ICL conditions. Furthermore, we demonstrate that LiFT yields larger gains for minority classes, meaningfully attends to recent demonstration history, and develops transferable longitudinal modeling capabilities beyond its training tasks

Figures & tables

Figure 1: Overview of LiFT: The longitudinal input construction stage (blue) groups observations by sequence, orders them chronologically, forms rolling histories, and converts them into structure with temporal and label metadata IDs. Training (green) follows a curriculum of increasing context length and difficulty: AnnoMI with (k=1) and a mean context token length of 142, LRS with (k=2) and 216, and TalkLife with (k=3) and (1.0 K ). The metadata IDs are embedded, projected, and added to the token representations through temporal–label conditioning. The instruction-tuned LLM is adapted using LoRA, together with a global classification head and a history-only auxiliary head. The joint objective (red) combines (A) prompt cross-entropy, (B) focal language-model loss, (C) global focal loss and (D) history-only focal loss.

Dataset	AnnoMI			LRS			TalkLife			Reddit (OOD)			CMV (OOD)
Shots	0	1	3	0	1	3	0	1	3	0	1	3	0	1	3
OLMo-1B (IFT)	.18	.18	.18	.35	.36	.37	.23	.22	.29	.14	.27	.15	.36	.36	.37
OLMo-1B (LiFT)	.29 ***	.34 ***	.36 ***	.45 ***	.47 ***	.53 ***	.22	.25 ***	.32 *	.30 ***	.32 ***	.32 ***	.40 ***	.38 **	.39 ***
OLMo-7B (IFT)	.21	.38	.45	.12	.13	.13	.03	.01	.06	.23	.22	.30	.48	.47	.49
OLMo-7B (LiFT)	.34 ***	.44	.45	.38	.58 ***	.59 ***	.30 ***	.31 ***	.32 ***	.30	.30 ***	.32	.60 ***	.61 ***	.62 ***
LLaMA-8B (IFT)	.21	.35	.40	.44	.46	.43	.07	.07	.18	.29	.29	.29	.47	.47	.46

Table 1: Red-shaded datasets are OOD. Bold indicates LiFT improvements based on the Macro-F1 scores. Superscripts report BH-corrected paired-bootstrap significance (

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

); † indicates a significant decrease. Three-seed results are provided in Table 15 . Section H.1 and Table 12 report shot-dependent gains and in-context amplification.

Figure 2: LiFT vs. IFT: Panels (a–f) report Macro-F1 averaged across model families for 0-, 1-, and 3-shot prompting. Per-model gains, aggregation details, and aggregated values are provided in Figure 5 , Section G , and Table 11 .

	AnnoMI			LRS			TalkLife			Reddit (OOD)			CMV (OOD)
Shots	0	1	3	0	1	3	0	1	3	0	1	3	0	1	3
IFT	.21	.38	.45	.12	.13	.13	.03	.01	.06	.23	.22	.30	.48	.47	.49
SFT	.29	.31	.33	.33	.35	.38	.07	.05	.06	.27	.24	.28	.50	.47	.51
LiFT w/o Temporal	.28	.32	.33	.31	.35	.38	.06	.05	.06	.27	.24	.29	.48	.48	.50
LiFT w/o Curriculum	.28	.32	.33	.31	.35	.38	.06	.05	.06	.26	.24	.29	.49	.48	.51
Reverse 1–2–3	.29	.33	.34	.33	.36	.39	.07	.06	.06	.27	.25	.30	.51	.49	.53

Table 2: OLMo-7B ablation: Each variant removes or modifies one LiFT component while keeping all other components unchanged. Full results are provided in Table 13 and Figure 9 .

Family	$\Delta$ F1	Wins/Sig.	McN. OR	DID
OLMo-1B	$+.090/.095$	$14/15;14\,(+1\,\mathrm{dec.})$	$1.65;12$	$+.016;5$
OLMo-7B	$+.184/.132$	$15/15;10$	$10.12;14$	$+.003;3$
LLaMA-8B	$+.062/.057$	$14/15\,(1\,\mathrm{tie});10$	$1.15;9$	$+.018;5$
Qwen-14B	$+.172/.153$	$14/15;14\,(+1\,\mathrm{dec.})$	$2.14;12$	$+.066;8$
Qwen-32B	$+.050/.064$	$15/15;9$	$3.14;12$	$+.004;9$
Overall	$+.112/.081$	$72/75;57\,(+2\,\mathrm{dec.})$	$2.14;59$	$+.021;30$

Table 3: Statistical summary of LiFT over IFT.

\Delta

F1 reports the mean/median LiFT–IFT macro-F1 across dataset–shot settings. Wins/Sig. reports positive wins and BH-adjusted significant paired-bootstrap gains. McN. OR reports the median McNemar odds ratio and number of significant tests. DID reports the mean few-shot difference-in-differences and number of significant positive effects.

Figure 3: LiFT/IFT under longer contexts. Macro-F1 on OOD datasets using Qwen-14B for 0–9 shots ( Appendix H.2 ).

Figure 4: Mechanistic analysis of LiFT versus IFT using OLMo-7B. (A) Probing deltas measure the change in label decodability, with positive values indicating that LiFT representations encode more label-relevant information than IFT. (B) Normalized history-recency attention measures the distribution of attention within the history, where larger values at smaller rel_t indicate a stronger focus on recent observations. (C) Activation-patching harm reports the reduction in Macro-F1 when corrupted <HIST> activations are inserted into an otherwise clean forward pass, indicating causal dependence on the learned history representations. (D) Routing shift by layer shows how attention from the prediction position is redistributed across input regions under LiFT relative to IFT. Together, these analyses connect improved representation of label-relevant information (A) , temporal attention and routing (B,D) , and causal reliance on history representations (C) .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Acronym	Definition
NLP	Natural Language Processing.
LLM	Large Language Model.
ICL	In-Context Learning, where a model performs a task using demonstrations provided in the input prompt without updating its parameters.
LiFT	Longitudinal Instruction Fine-Tuning, the proposed framework for improving longitudinal reasoning and in-context learning.
IFT	Vanilla instruction-tuned checkpoint.
SFT	Standard supervised fine-tuning baseline.

Appendix

Table 4: Acronyms and abbreviations used throughout the paper.

Symbol	Definition
$H$	Temporally ordered history associated with the current prediction instance.
$H=\{(x_{1},t_{1}),\ldots,(x_{m},t_{m})\}$	Sequence of $m$ historical observations and their corresponding timestamps, ordered such that $t_{1}\leq\cdots\leq t_{m}$ .
$x_{i}$	The $i$ -th observation in a longitudinal sequence.
$t_{i}$	Timestamp or temporal position associated with observation $x_{i}$ .
$m$	Number of observations included in the historical context.
$(x_{m+1},t_{m+1})$	Current observation and its timestamp, for which a prediction is required.

Appendix

Table 5: Mathematical notation used in the LiFT formulation, prompt construction, training objective, and curriculum algorithm.

Term or symbol	Definition
hist_mean	Mean hidden-state representation computed over tokens in the <HIST> span. It measures how much task-relevant information is encoded in the historical context.
fewshot_mean	Mean hidden-state representation computed over tokens belonging to the few-shot demonstrations.
curr_mean	Mean hidden-state representation computed over tokens in the current observation.
last_curr	Hidden-state representation of the final decision-facing token in the current-observation span.
$\mathrm{rel\_t}$	Relative temporal bin assigned to an item within the history. Smaller values represent items closer to the current observation.
$\Delta_{\mathrm{probe}}$	Difference in probing performance between LiFT and the corresponding IFT model for a given layer and prompt region.

Appendix

Table 6: Notation used in the mechanistic interpretability analyses.

Symbol	Definition
$\Delta F1$	Difference in macro-F1 between LiFT and the corresponding IFT model.
$L_{i}$	Indicator that the IFT model predicts example $i$ correctly.
$L_{i}$	Indicator that the LiFT model predicts example $i$ correctly.
$\mathbb{I}[\cdot]$	Indicator function.
$b_{01}$	Number of examples incorrect under IFT but correct under LiFT.
$b_{10}$	Number of examples correct under IFT but incorrect under LiFT.

Appendix

Table 7: Notation used in the statistical analyses.

\mathcal{L}=\lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}}+\lambda_{\mathrm{out}}\mathcal{L}_{\mathrm{focalLM}}+\lambda_{\mathrm{cls}}\mathcal{L}_{\mathrm{focalCls}}+\lambda_{\mathrm{hist}}\mathcal{L}_{\mathrm{histCls}}

Appendix

Algorithm 2 LiFT Trainer: Joint Longitudinal Fine-Tuning

Dataset	Domain and Sequence	Task Definition	Labels	Key Characteristics	Role
AnnoMI	Motivational-interviewing dialogue	Given the preceding therapist–client dialogue, classify the motivational language expressed in the current client utterance.	Change , Sustain , Neutral	44 topics; 6,725 utterances; structured and relatively short contexts (mean: 142 tokens).	Train
LRS	Twitter rumour timelines	Given the chronological history of a rumour conversation, predict whether the stance changes at the current post.	Switch , Non-Switch	274 timelines; 4,239 posts; medium-length, noisy social-media sequences (mean: 216 tokens).	Train
TalkLife	Mental-health user timelines	Given a user’s previous posts and current post, identify whether their emotional or mental state changes or escalates.	IS , IE , O	500 timelines; 18,702 posts; long contexts with sparse and often implicit changes (mean: 1,049 tokens).	Train
Reddit MoC	Reddit mental-health timelines	Detect switches, escalations, or stability in a user’s mental state across chronologically ordered Reddit posts.	IS , IE , O	255 timelines; 6,195 posts; longest and noisier user histories (mean: 2,583 tokens).	OOD only
CMV	Reddit opinion-change discussions	Given a reply and the author’s previous same-topic contributions, predict whether the reply changes the author’s view.	1: Change , 0: No change	9,456 author-topic timelines; 48,570 replies; binary and highly sparse change events (mean: 786 tokens).	OOD only

Appendix

Table 8: Overview of the longitudinal datasets. Green rows indicate datasets used for LiFT training and in-domain (ID) evaluation with an 80/20 split. Red rows indicate datasets used exclusively for out-of-distribution (OOD) evaluation.

	LRS	TalkLife	AnnoMI	Reddit	CMV
Dataset Overview
Split Type	timelines	timelines	topics	timelines	timeline
Total Timelines	274	500	44	255	9,456
Train Timelines	219	400	35	–	–
Test Timelines	55	100	9	Full Set	Full Set
Total Posts	4,239	18,702	6,725	6,195	48,570

Appendix

Table 9: Comprehensive dataset statistics across all datasets.

Model	Context	0	1	3
OLMo-7B	IFT (Author-all)	.487	.478	.513
	LiFT (Author-all)	.543	.545	.564
	IFT (Conv.)	.482	.471	.506
	LiFT (Conv.)	.488	.479	.514
	IFT (Author-topic)	.482	.470	.486
	LiFT (Author-topic)	.601	.606	.618

Appendix

Table 10: Macro-F1 on CMV with different context sources.

Figure 5: Per-model Macro-F1 gains of LiFT over IFT across datasets and shot settings. Each cell reports

\Delta_{m,d,s}=F^{L}_{m,d,s}-F^{I}_{m,d,s}

, where positive values indicate improved performance with LiFT.

	IFT			LiFT			Gain
Dataset	0-shot	1-shot	3-shot	0-shot	1-shot	3-shot	0-shot	1-shot	3-shot
AnnoMI	.292	.341	.357	.389	.479	.517	+.096	+.138	+.161
LRS	.332	.342	.327	.428	.486	.504	+.096	+.144	+.178
TalkLife	.158	.125	.186	.267	.277	.318	+.110	+.152	+.132
Reddit (OOD)	.197	.252	.251	.289	.339	.361	+.092	+.087	+.109
CMV (OOD)	.494	.509	.516	.540	.569	.590	+.047	+.060	+.074

Appendix

Table 11: Aggregated Macro-F1 values used to construct Figure 5 . For each dataset and shot setting, IFT and LiFT scores are averaged across the five model families reported in Table 1 . The gain is computed as

\Delta_{d,s}=F^{\mathrm{LiFT}}_{d,s}-F^{\mathrm{IFT}}_{d,s}

	$0\rightarrow 1$ shot			$1\rightarrow 3$ shots
Model	$\Delta^{I}$	$\Delta^{L}$	Amp.	$\Delta^{I}$	$\Delta^{L}$	Amp.
OLMo-1B	$+.027$	$+.018$	$-.009$	$-.006$	$+.034$	$+.040$
OLMo-7B	$+.027$	$+.064$	$+.037$	$+.043$	$+.012$	$-.031$
LLaMA-8B	$+.035$	$+.054$	$+.019$	$+.024$	$+.041$	$+.018$
Qwen-14B	$-.022$	$+.073$	$+.094$	$-.010$	$+.028$	$+.038$
Qwen-32B	$+.028$	$+.027$	$-.001$	$+.018$	$+.026$	$+.008$

Appendix

Table 12: Incremental in-context learning improvement of LiFT relative to IFT.

\Delta^{I}

and

\Delta^{L}

denote the changes in Macro-F1 for IFT and LiFT, respectively, when moving from zero to one demonstration and from one to three demonstrations. ICL amplification is defined as

\operatorname{Amp}=\Delta^{L}-\Delta^{I}

. A positive value indicates that LiFT obtains a greater incremental benefit from additional demonstrations than IFT. Values are averaged across the five datasets.

Figure 6: OLMo-7B under longer OOD contexts. Macro-F1 of IFT and LiFT on Reddit and CMV across 0–9 shots. LiFT consistently outperforms Base on both datasets. On Reddit, performance remains stable and improves slightly at 7–9 shots, whereas on CMV both methods decline beyond 3 shots, but LiFT retains a clear advantage.

	AnnoMI			LRS			TalkLife			Reddit (OOD)			CMV (OOD)
Shots	0	1	3	0	1	3	0	1	3	0	1	3	0	1	3
OLMo-1B (IFT)	.178	.178	.178	.354	.364	.371	.225	.217	.290	.136	.268	.150	.361	.362	.369
SFT	.219	.257	.279	.359	.325	.354	.052	.041	.044	.260	.216	.238	.337	.342	.369
LiFT w/o Temporal	.215	.259	.278	.353	.326	.356	.051	.042	.044	.236	.219	.261	.325	.351	.358
LiFT w/o Curriculum	.214	.257	.279	.352	.327	.355	.048	.042	.045	.235	.218	.262	.332	.347	.360
LiFT w/o History Loss	.217	.255	.291	.339	.345	.351	.047	.041	.050	.227	.260	.234	.371	.344	.339

Appendix

Table 13: Full ablation study of the LiFT framework across five model scales ( OLMo-1B , OLMo-7B , LLaMA-8B , Qwen-14B , and Qwen-32B ). We compare the IFT model, SFT baseline, full LiFT model, and variants removing temporal conditioning, curriculum learning, and the history-only auxiliary loss. Results are reported as Macro-F1 across five datasets and 0-, 1-, and 3-shot settings. Bold indicates the highest Macro-F1 within each model, dataset, and shot setting; tied values are not bolded.

Component	Details
Models and compute
Models	OLMo-1B, OLMo-7B, LLaMA-3.1-8B, Qwen2.5-14B, and Qwen2.5-32B
Model Checkpoints	Exact Hugging Face model identifiers and revisions are recorded in the released configuration files
Hardware	NVIDIA A100 (80GB), H100, and GH200 (120GB) GPUs
# GPUs	2–3 depending on model size; Qwen2.5-32B uses 2 GPUs
Training Method	LoRA-based parameter-efficient fine-tuning; where backbone parameters remain frozen

Appendix

Table 14: LiFT compute and reproducibility configuration. The table summarizes the principal training, inference, evaluation, and resource settings.

AnnoMI

LRS

TalkLife

Reddit (OOD)

CMV (OOD)

Shots

OLMo-1B (IFT)

.178 (

\pm

.000)

.178 (

\pm

.000)

.178 (

\pm

.000)

.354 (

\pm

.000)

.364 (

\pm

.000)

.371 (

\pm

.000)

..225 (

\pm

.000)

.217 (

\pm

.000)

.290 (

\pm

.000)

.136 (

\pm

.024)

.268 (

\pm

.015)

.150 (

\pm

.025)

.361 (

\pm

.000)

.362 (

\pm

.000)

.369 (

\pm

.000)

OLMo-1B (LiFT)

.290 *** (

\pm

.000)

.335 *** (

\pm

.000)

.363 *** (

\pm

.000)

.449 *** (

\pm

.003)

.473 *** (

\pm

.000)

.528 *** (

\pm

.000)

.224 (

\pm

.016)

.254 *** (

\pm

.000)

.321 * (

\pm

.000)

.301 *** (

\pm

.000)

.317 *** (

\pm

.000)

.323 *** (

\pm

.000)

.400 *** (

\pm

.000)

.377 ** (

\pm

.000)

.389 *** (

\pm

.000)

OLMo-7B (IFT)

.207 (

\pm

.000)

.379 (

\pm

.000)

.446 (

\pm

.000)

.124 (

\pm

.174)

.128 (

\pm

.000)

.125 (

\pm

.000)

.029 (

\pm

.000)

.008 (

\pm

.000)

.064 (

\pm

.000)

.227 (

\pm

.023)

.219 (

\pm

.019)

.298 (

\pm

.014)

.482 (

\pm

.000)

.470 (

\pm

.000)

.486 (

\pm

.000)

OLMo-7B (LiFT)

.339 *** (

\pm

.000)

.443 (

\pm

.000)

.448 (

\pm

.000)

.380 (

\pm

.000)

.584 *** (

\pm

.000)

.587 *** (

\pm

.000)

.298 *** (

\pm

.000)

.305 *** (

\pm

.000)

.324 *** (

\pm

.000)

.298 (

\pm

.000)

.300 *** (

\pm

.000)

.319 (

\pm

.000)

.601 *** (

\pm

.000)

.606 *** (

\pm

.000)

.618 *** (

\pm

.000)

LLaMA-8B (IFT)

.207 (

\pm

.000)

.352 (

\pm

.000)

.400 (

\pm

.000)

.437 (

\pm

.000)

.460 (

\pm

.000)

.430 (

\pm

.000)

.067 (

\pm

.000)

.073 (

\pm

.000)

.180 (

\pm

.000)

.292 (

\pm

.000)

.291 (

\pm

.000)

.291 (

\pm

.000)

.466 (

\pm

.000)

.468 (

\pm

.000)

.462 (

\pm

.000)

Appendix

Table 15: Detailed results corresponding to Table 1 . Macro-F1 is reported as mean

\pm

standard deviation over three seeds. Values are shown to three decimal places; therefore, some standard deviations round to .000. Bold indicates the higher Macro-F1 within each IFT–LiFT comparison. Significance markers are defined in Table 1 .

Model family	$\Delta$ Macro-F1	Macro-F1 tests	Sign test	McNemar	Minority effect	Few-shot DID
OLMo-1B	$+.100/+.106$	$15/15$ wins; $15$ sig.; adj. $\tilde{p}_{boot}=0.000$	$15/15$ , $p=6.1{\times}10^{-5}$	$\widetilde{\mathrm{OR}}=1.65$ ; $12/15$ sig.; adj. $\tilde{p}=6.3{\times}10^{-9}$	$\Delta$ F1 $-.121/-.099$ ; $1/15$ sig.+; recall $0/15$ sig.+	$+.011/+.010$ ; $5/15$ sig.+; adj. $\tilde{p}=.019$
OLMo-7B	$+.184/+.132$	$15/15$ wins; $10$ sig.; adj. $\tilde{p}_{boot}=0.000$	$15/15$ , $p=6.1{\times}10^{-5}$	$\widetilde{\mathrm{OR}}=10.12$ ; $14/15$ sig.; adj. $\tilde{p}=2.0{\times}10^{-34}$	$\Delta$ F1 $+.089/+.098$ ; $10/15$ sig.+; recall $7/15$ sig.+	$+.027/+.005$ ; $3/15$ sig.+; adj. $\tilde{p}=.112$
LLaMA-8B	$+.062/+.057$	$14/15$ wins (one tie); $10$ sig.; adj. $\tilde{p}_{boot}=0.000$	$14$ wins and one tie, $p=1.2{\times}10^{-4}$	$\widetilde{\mathrm{OR}}=1.15$ ; $9/15$ sig.; adj. $\tilde{p}=9.2{\times}10^{-4}$	$\Delta$ F1 $+.023/+.011$ ; $7/15$ sig.+; recall $8/15$ sig.+	$+.022/+.024$ ; $5/15$ sig.+; adj. $\tilde{p}=.066$
Qwen-14B	$+.172/+.153$	$14/15$ wins; $14$ sig.; adj. $\tilde{p}_{boot}=0.000$	$14/15$ , $p=9.8{\times}10^{-4}$	$\widetilde{\mathrm{OR}}=2.14$ ; $12/15$ sig.; adj. $\tilde{p}=1.5{\times}10^{-16}$	$\Delta$ F1 $+.194/+.218$ ; $11/15$ sig.+; recall $8/15$ sig.+	$+.075/+.043$ ; $8/15$ sig.+; adj. $\tilde{p}=0.000$
Overall	$+.130/+.098$	$58/60$ wins (one tie); $49$ sig.; adj. $\tilde{p}_{boot}=0.000$	$58$ wins, one tie, and one loss, $p=2.1{\times}10^{-16}$	$\widetilde{\mathrm{OR}}=1.89$ ; $47/60$ sig.; adj. $\tilde{p}=2.6{\times}10^{-16}$	$\Delta$ F1 $+.046/+.018$ ; $29/60$ sig.+; recall $23/60$ sig.+	$+.034/+.019$ ; $21/60$ sig.+; adj. $\tilde{p}=.064$

Appendix

Table 16: Summary of statistical tests across model families.

\Delta

Macro-F1 reports mean/median LiFT–IFT differences across dataset–shot conditions. “Macro-F1 tests” reports positive LiFT wins, Benjamini–Hochberg significant paired-bootstrap gains, and median adjusted bootstrap

p

-value. McNemar reports the median corrected-vs-broken odds ratio, number of significant tests, and median adjusted

p

-value. Minority effect reports mean/median minority-vs-rest F1 difference, significant positive minority-F1 tests, and significant positive minority-recall tests. DID measures whether few-shot demonstrations improve LiFT more than IFT. “sig.” denotes Benjamini–Hochberg adjusted

p<0.05

Panel of Figure 10 — Figure 10: Prompt templates used for TalkLife timeline mood change detection under 0-shot, 1-shot, and 3-shot prompting settings.

Panel of Figure 11 — Figure 11: Prompt templates used for AnnoMI client-talk classification under 0-shot, 1-shot, and 3-shot prompting settings using topic-level timelines.

Panel of Figure 12 — Figure 12: Prompt templates used for LRS stance switch classification under 0-shot, 1-shot, and 3-shot prompting settings.

Panel of Figure 13 — Figure 13: Prompt templates used for Reddit timeline mood change classification under 0-shot, 1-shot, and 3-shot prompting settings.

Panel of Figure 14 — Figure 14: Prompt templates used for CMV conversation timeline view-change classification under 0-shot, 1-shot, and 3-shot prompting settings.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 402 papers

Feb 20, 2025

LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-context performance of arbitrary short-context LLMs by dynamically adapting their parameters to each long input. Instead of endlessly extending context windows to fit longer inputs in context, LIFT stores and absorbs the input in parameters. By fine-tuning long inputs into parameters, LIFT enables short-context LLMs to answer questions even when required information is absent from the inference context, avoiding the quadratic input-length complexity of standard long-context models. Rather than simple continued pretraining on new long contexts, LIFT uses carefully designed LLM-generated synthetic tasks to enhance comprehension beyond memorization. To offset fine-tuning overhead, we design a highly optimized pipeline that reduces Time to First Token (TTFT) to under 10 seconds for 8k context. We further analyze LIFT's strengths and limitations, discuss large-scale deployment feasibility, and highlight future research directions. Implementation is open-sourced at https://github.com/MuLabPKU/LIFT.

Yansheng Mao, Yufei Xu, Jiaqi Li +5

Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI

Long-Context QA LLM Fine-Tuning Long-Context Modeling Test-Time Training

Ask PDF

Oct 16, 2026

Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks

In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates. Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support. We develop a theoretical framework under mild assumptions that separates these two forms of distribution shift. For target tasks represented in pretraining, we show that demonstrations can identify the task and reduce sensitivity to pretrain-supported prompt templates as their number increases. For an unsupported instruction, we show that under a fixed model-induced reference extension, demonstration evidence can dominate an

N

-independent instruction shift while preserving the exponential identification rate. We then use the same ICL framework as an interpretive lens for long-horizon target tasks that are not represented in pretraining but admit a plan that decomposes them into pretraining-supported subtasks. Applying the supported-task analysis step by step yields a standard accumulated-error bound and clarifies how invoking familiar subtasks along an intermediate-state trajectory can avoid directly selecting the unsupported complete task. Together, our results characterize when demonstrations, instructions, and task decomposition can support ICL beyond the corresponding pretraining supports.

Xuhan Tong, Yuchen Zeng, Jiawei Zhang

Department of Computer Sciences, University of Wisconsin–Madison. · Microsoft Research.

In-Context Learning

Ask PDF

May 12, 2026

Learning, Fast and Slow: Towards LLMs That Adapt Continually

Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating model parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. Inspired by how human learning works at multiple time scales (e.g., System 1 vs 2), we propose a fast-slow learning framework for LLMs, instantiated with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while RL stalls.

Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal +6

UC Berkeley · Eragon · Mila +3

Continual Learning for LLMs RL for Language Model Reasoning In-Context Learning Catastrophic Forgetting

Ask PDF

Abstract

In-Context Learning

Explore similar work

LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks

Learning, Fast and Slow: Towards LLMs That Adapt Continually