TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

Jichao Jiang; Cristian McGee; El Houcine Bergou; Hanqin Cai; Aritra Dutta

cs.LGOct 1, 2026

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

Authors: Jichao Jiang, Cristian McGee, El Houcine Bergou, Hanqin Cai, Aritra Dutta

Organizations: University of Central Florida · Mohammed VI Polytechnic University

Abstract

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized $1\to1$ operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by $174\times$ relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by $2.9\times$ (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.

Figures & tables

Figure 1: Performance–memory overview of TACO and other optimizers. Left: Pareto frontier memory-accuracy trade-off for fine-tuning OPT-13B on the SST-2 dataset on an 80 GB H100 GPU. TACO lies at the low-memory end of the Pareto frontier, achieving 94.22% accuracy with 27.54 GB of peak training memory. The dashed line connects the Pareto-optimal methods. The vertical line marks the 80 GB H100 memory limit. Adam achieves 95.3% accuracy with 254.8 GB of peak memory. Right: Peak GPU memory decomposition of TACO on OPT-30B across 8 downstream tasks. The bars show model state, optimizer state, and other transient memory across all eight downstream tasks. TACO’s optimizer state occupies only 0.267 GB, less than 0.5% of peak memory on each task, while the transient memory accounts for the variation across tasks.

Panel of Figure 2 — Figure 2: Illustrative workflow of the memory-efficient practical implementation of TACO. We use $\mathcal{H}_{2}$ for illustration; in our experiments, we use $\mathcal{H}_{16}$ . In this example, we highlight the largest- and second-largest-magnitude elements for $\mathcal{H}_{2}$ in each column. Here, $G_{t}$ is the dense gradient via autograd , $M_{t}$ is obtained via $\mathcal{H}_{2}(G_{t})$ by retaining the largest- and second-largest-magnitude elements in each column, and finally, the operator $\mathcal{S}$ is used, which keeps the sign of the largest-magnitude element in each column. Only the sparse gradient history $M_{t}$ persists across iterations.

Figure 3: OPT-13B performance and peak-memory comparison across downstream tasks. Upper bars report test accuracy or F1; lower mirrored bars report peak GPU memory. OOM denotes configurations that do not fit within the available GPU memory. Across all tasks, TACO consistently requires the least GPU memory while maintaining competitive performance; see Table 3 in Appendix B for full results.

Figure 4: Efficiency comparison in terms of optimization trajectory, peak memory decomposition, and training throughput for fine-tuning OPT-13B on SST-2. For the Train loss vs. Steps plot, we show only the first 500 steps, as all optimizers plateaued; the shaded region shows

\pm

sample standard deviation across five training seeds; see Table 4 in Appendix B for full results.

Figure 5: Scaling behavior of TACO across the OPT and Qwen3 model families. We plot the test performance against peak GPU memory as model size increases; see Table 5 in Appendix B for full results.

Figure 6: Transfer across model architectures and learning rate stability across model scales. Left: TACO performance and peak GPU memory across Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral-24B on SST-2, RTE, and BoolQ, showing consistent memory efficiency with task-dependent performance. Right: Learning rate sweep on RTE for OPT-1.3B, OPT-6.7B, OPT-13B, and OPT-30B. All four models attain their best validation accuracy at

3\times 10^{-5}

, supporting learning rate transfer across model scales.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Setting
Fine-tuning	Full-parameter
Model precision	BF16
Maximum sequence length	1,024
Gradient accumulation	1
Learning rate	3e-5
Learning rate schedule	Linear

Appendix

Table 1: Shared experimental configuration. All experiments use this setting unless otherwise specified.

Experiment	Model and task	Optimizer	Experiment specific hyperparameters
E1 Optimizer comparison Fix model, task	Model: OPT-13B Tasks: SST-2, RTE, BoolQ, COPA, CB, MultiRC, SQuAD, DROP	TACO	–
		AdamW8bit	LR = 3e-6 ; 8-bit optimizer states; FP32 embedding; weight decay $=0.01$
		Adafactor	LR = 1e-4 ; $\beta_{2}$ decay $=-0.8$ ; weight decay $=0$
		APOLLO-Mini	LR = 3e-6 ; rank $=1$ ; projection gap $=200$ ; tensor-wise scale $=128$
		FlashAdamW	LR = 1e-6 ; BF16 weights; 8-bit weight correction; 8-bit moments; weight decay $=0.01$
		GaLore	LR = 3e-5 ; rank $=128$ ; projection gap $=200$ ; scale $=0.25$ ; weight decay $=0$

Appendix

Table 2: Experiment configurations for E1-E4. E1 compares optimizers with the model and tasks fixed. E2 studies TACO across model scales. E3 contains the ablation studies. E4 evaluates transfer across model architectures. Settings not explicitly listed follow Table 1 . Note that TACO uses uniform hyperparameters in Table 1 , except the ablation study category, E3, where only explicitly listed hyperparameters are varied.

	Classification			Multiple Choice			Generation
Method	SST-2	RTE	BoolQ	COPA	CB	MultiRC	SQuAD	DROP
TACO	94.2	80.8	71.5	89.4	87.1	61.2	82.6	30.8
AdamW8bit	95.3	83.5	62.1 (2/5)	88.2	71.8	OOM	88.1	OOM
Adafactor	95.5	87.3	82.0	87.0	90.0	75.1	88.6	37.7
APOLLO-Mini	93.0	83.5	77.8	87.0	80.7	64.2	88.8	36.7
FlashAdamW	95.5	86.3	64.8 (4/5)	86.8	85.7	57.5	88.4	39.8

Appendix

Table 3: OPT-13B performance comparison across downstream tasks. SST-2, RTE, BoolQ, COPA, and CB report accuracy; MultiRC reports F1A; SQuAD and DROP report F1. For entries marked

(n/5)

, only

n

of the five runs complete successfully and the remaining runs fail due to OOM.

Optimizer	Steps	State Mem. $\downarrow$	Peak Mem. $\downarrow$	Runtime $\downarrow$	Throughput $\uparrow$	Acc. $\uparrow$	Best Val. Step $\downarrow$
		(GB)	(GB)	(minutes)	(tokens/s)	(%)
TACO	2K	0.159	27.54	14.92	275.8	94.22	1240
Adafactor	2K	0.013	52.59	11.77	349.7	95.50	840
AdamW8bit	2K	27.708	80.60	15.74	261.3	95.25	1380
APOLLO-Mini	2K	1.095	53.67	8.95	460.1	92.98	940
FlashAdamW	2K	40.167	67.68	7.55	546.7	95.48	940

Appendix

Table 4: OPT-13B fine-tuning efficiency on SST-2. Best Val. Step reports the mean checkpoint selected by validation performance.

Family	Model Size	SST-2		RTE		BoolQ
		Acc.	Mem. (GB)	Acc.	Mem. (GB)	Acc.	Mem. (GB)
OPT	1.3B	92.9	3.4	69.6	4.0	63.7	5.5
	6.7B	94.4	14.8	74.0	15.5	68.1	17.3
	13B	94.7	27.6	78.5	28.6	72.3	31.0
	30B	93.3	62.5	79.5	64.4	67.1	68.2
Qwen3	8B	93.9	19.4	88.8	20.6	86.3	24.5

Appendix

Table 5: Scaling behavior of TACO across the OPT and Qwen3 model families.

Task	$\bm{T=2\mathrm{K}}$	$\bm{T=4\mathrm{K}}$	$\bm{T=8\mathrm{K}}$	$\bm{T=16\mathrm{K}}$	Change
BoolQ	68.6	72.6	73.3	75.1	$+6.5$ pp
RTE	72.6	76.5	76.7	76.9	$+4.3$ pp
SQuAD	78.6	80.0	80.8	80.7	$+2.1$ F1

Appendix

Table 6: TACO optimizer step budget ablation on OPT-1.3B with

T\in\{2\mathrm{K},4\mathrm{K},8\mathrm{K},16\mathrm{K}\}

. The change is measured from

T=2\mathrm{K}

T=16\mathrm{K}

Task	$\bm{\mu=0}$	$\bm{\mu=0.95}$	Change
SST-2	93.7	93.9	$+0.3$ pp
RTE	71.2	72.6	$+1.3$ pp
BoolQ	66.1	68.6	$+2.5$ pp
COPA	78.0	79.0	$+1.0$ pp
CB	79.2	86.3	$+7.1$ pp
MultiRC	53.5	62.5	$+8.9$ F1A

Appendix

Table 7: Effect of gradient history EMA decay factor

\mu

on OPT-1.3B. SST-2, RTE, BoolQ, COPA, and CB report accuracy; MultiRC reports F1A; SQuAD and DROP report F1.

	Dense	$\bm{k=64}$	$\bm{k=32}$	$\bm{k=16}$	$\bm{k=8}$	$\bm{k=4}$	$\bm{k=2}$	$\bm{k=1}$
BoolQ	68.1	66.8	67.9	68.6	67.5	65.5	65.8	66.3
RTE	69.7	70.9	71.0	72.6	70.8	70.3	69.3	69.7
SQuAD	79.4	79.4	79.4	78.6	78.9	78.6	77.7	78.4
State memory (MB)	1317.7	145.4	74.0	38.3	20.4	11.5	7.0	4.8

Appendix

Table 8: TACO heavy hitter threshold

k

ablation on OPT-1.3B.

Task	Top-1 (TACO)	Top-2	Top-4	Top-8	Top-16	Top-32
BoolQ	67.9	66.6	67.8	61.9	61.5	61.5
RTE	71.0	70.4	69.3	53.6	56.1	47.1
SQuAD	79.4	79.0	76.8	76.0	73.0	70.7

Appendix

Table 9: TACO update Top-

k

ablation on OPT-1.3B with heavy hitter threshold

k=32

	Qwen3-14B				Qwen3-32B
	SST-2		RTE		SST-2		RTE
Batch	Acc.	Mem. (GB)	Acc.	Mem. (GB)	Acc.	Mem. (GB)	Acc.	Mem. (GB)
2	$94.1\pm 0.9$	33.1	$89.5\pm 1.2$	33.8	$92.3\pm 2.6$	69.3	$89.4\pm 1.9$	70.3
4	$94.2\pm 1.0$	33.3	$89.2\pm 1.3$	34.8	$93.3\pm 1.6$	69.6	$\mathbf{90.1}\pm 0.8$	71.5
8	$93.9\pm 0.4$	33.6	$89.3\pm 1.9$	36.6	$93.7\pm 0.5$	70.0	$89.4\pm 0.9$	74.0
16	$94.5\pm 0.6$	34.3	$\mathbf{89.8}\pm 0.8$	43.1	$93.1\pm 0.9$	71.0	$89.3\pm 0.6$	81.4

Appendix

Table 10: TACO batch size ablation on Qwen3-14B and Qwen3-32B. Accuracy is reported as mean

\pm

sample standard deviation, and memory as peak GPU memory.

Figure 7: Every evaluated model and dataset configuration attains its best validation performance at

3\times 10^{-5}

, empirically supporting the

\mu

P scaling result in Theorem 4

Explore similar work

May 9, 2026cs.LG

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers

Training large language models requires optimization algorithms that are not only statistically effective, but also computationally and memory efficient at extreme scale. Although Adam remains the dominant optimizer for large-scale language-model pretraining and fine-tuning, recent work has revisited nearly every component of the optimization stack: adaptive moment estimation, decoupled weight decay, memory footprint, curvature approximation, sign-based updates, large-batch stability, low-rank gradient structure, and matrix-wise orthogonalized updates. This survey reviews optimizer design for large language models through a systems-and-optimization lens. We organize the literature into classical first-order optimizers, adaptive optimizers, memory-efficient variants, second-order and curvature-aware methods, sign-based and discovered optimizers, low-rank and projection-based methods, and matrix-based optimizers such as Muon. We also discuss benchmarking methodology, including hyperparameter fairness, scale dependence, wall-clock efficiency, token efficiency, memory overhead, and downstream evaluation. We argue that optimizer research for LLMs is entering a new phase: moving from single-algorithm speedup claims toward rigorous, scale-aware comparisons that jointly evaluate convergence, stability, memory, and implementation complexity.

Aditya Ranganath

Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA

Optimizer Design Large Language Model Training Large Language Model Pretraining Bounded-Memory Low-Rank Structure

Ask PDF

Oct 5, 2026cs.LG

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while constraints maintained throughout training can limit task-specific adaptation and raise the attainable loss floor. We introduce ORCA (Orthogonal Regularization, Cooled After), a minimal optimizer intervention that applies strong but temporary soft orthogonality regularization early in training, then removes it. This allows the weights to benefit from a broader spectrum early on and adapt freely afterward. Across LLaMA, Qwen3, and fine-grained mixture-of-experts models ranging from 130M to 8B parameters, ORCA achieves lower final validation loss than Muon. Its loss reduction relative to Muon matches or exceeds Muon's reduction relative to Adam. Ablations support the early-shaping, later-release design. Further, ORCA requires no architectural changes and adds minimal overhead.

Yuanshi Liu, Boyuan Jiang, Liang Hou +4

School of Intelligence Science and Technology, Peking University · Kling Team

Topic pending

Ask PDF

Jul 10, 2026cs.LG

Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.

Ivan Ilin, Philip Zmushko, Peter Richtárik

KAUST

Parameter-Efficient Fine-Tuning Methods Model Fine-Tuning Tuning Saliency

Ask PDF