Repairing Speech Access After Text Updates with Frozen Affine Fields

Ming-Hao Hsu; Yuxiang Wang; Yingda Shen; Zhizheng Wu

cs.CLOct 14, 2026

Repairing Speech Access After Text Updates with Frozen Affine Fields

Authors: Ming-Hao Hsu, Yuxiang Wang, Yingda Shen, Zhizheng Wu

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen, China

Abstract

Speech-capable language models inherit most of their knowledge from text-rich language backbones, and continued model maintenance is likewise dominated by text updates. Knowledge acquired through a text update can nevertheless remain difficult to access when the same query is spoken. We formulate this mismatch as post-update speech access repair, where the knowledge-bearing checkpoint stays frozen and a detachable operator repairs speech-only access in one model pass. We introduce Affine Fields for Interface Repair across Modalities (AFFIRM), which first learns and freezes a base correction for speech hidden states, then adds query-specific corrections with zero mean over the operator-training queries. Paired speech and text states provide the coordinates that condition these corrections on each query. On balanced synthetic-speech PopQA, paired comparisons do not establish a clear advantage for AFFIRM over global SALSA steering or low-rank ReFT interventions under the same frozen single-state constraint. On recorded SLUE-SQA-5 questions with text passages, a full-scale PopQA-fitted AFFIRM operator transfers without refitting and improves ROUGE-L by 0.0941 over its paired No Repair baseline. Low-rank ReFT interventions also transfer positively, whereas the matched SALSA variant falls 0.0707 below its paired baseline. These results show that text acquisition and speech access can diverge after an update and should be verified and repaired separately.

Figures & tables

Panel of Figure 1 — Figure 1: Decoupled knowledge acquisition and speech access repair. (a) Text-only acquisition can leave a speech-access gap. (b) Affirm combines a base correction (shared access parent) with a query-specific correction (centered route field) and edits the final prompt state with the acquired checkpoint frozen. The example shows recorded outputs for one PopQA test question held out from operator fitting, with Affirm-Direct as the repair.

	Balanced PopQA		TOFU		SLUE no-refit
Method	RL	EM	RL	EM	Span	RL	EM
No Repair	0.2736	0.2189	0.5003	0.1000	0.4063	0.4435	0.2344
DiReFT ( Wu et al., 2024 )	0.5945	0.5821	0.5117	0.1075	0.4609	0.4981	0.2734
LoReFT ( Wu et al., 2024 )	0.5995	0.5920	0.5179	0.1138	0.4766	0.5209	0.3047
L2S ( Parekh et al., 2025 )	0.5361	0.5274	0.5107	0.1000	0.3906	0.4261	0.2656
SALSA ( Yegorova et al., 2026 )	0.6119	0.6020	0.5237	0.1150	0.3203	0.3730	0.2656

Table 1: Post-update factual access across source repair, author-disjoint evaluation, and no-refit recorded-speech transfer. PopQA reports the balanced test, TOFU reports 800 author-disjoint questions after TOFU fitting, and SLUE reports 256 recorded questions with text passages after PopQA-20 fitting without target-task training examples or strength retuning. RL denotes ROUGE-L recall, EM denotes normalized exact match, and Span denotes SLUE answer-span accuracy. All rows use one speech-only pass without a text-question donor, and Affirm-Direct edits decoder block 20. The displayed No Repair row is paired with Affirm and ReFT. On SLUE, L2S and SALSA use their separately generated No Repair outputs for the paired differences reported in the text and Appendix Table 9 .

	Balanced PopQA		TOFU		SLUE no-refit
Method	RL	EM	RL	EM	Span	RL	EM
No Repair	0.2769	0.2239	0.4996	0.0988	0.4102	0.4437	0.2422
Angle ( Xiang et al., 2025 )	0.5398	0.4925	0.6547	0.2550	0.4219	0.4619	0.2227
Length ( Xiang et al., 2025 )	0.2794	0.2239	0.5050	0.1025	0.4141	0.4476	0.2344

Table 2: Donor-assisted controls under a relaxed inference condition. Angle and Length modify the speech state using the paired text-question state at inference. No Repair reports the outputs generated alongside these controls on each dataset. RL, EM, and Span follow the definitions in Table 1 .

	TOFU		SLUE no-refit
Method	RL	EM	Span	RL	EM
Affirm-Direct (block 4)	0.5588	0.1425	0.5000	0.5335	0.3750
Affirm-Expert (block 4)	0.5798	0.1888	0.4805	0.5275	0.3320

Table 3: Parent parameterization affects the complete operator’s source and transfer performance. Affirm-Direct and Affirm-Expert combine the centered route field with the direct and expert parents, respectively. Both rows edit decoder block 4. TOFU reports RL and EM on 800 author-disjoint questions after TOFU fitting. SLUE reports answer-span accuracy (Span), RL, and EM on 256 recorded questions after the corresponding PopQA-20-fitted operators transfer without refitting.

	PopQA-20		PopQA-60		TOFU
Method	RL	EM	RL	EM	RL	EM
No Repair	0.1787	0.1500	0.1787	0.1500	0.5003	0.1000
+ Direct Parent	0.3156	0.2986	0.3269	0.3087	0.5252	0.1138
+ Route Field	0.3212	0.3038	0.3379	0.3203	0.5265	0.1213
Affirm-Direct	0.3353	0.3137	0.3493	0.3270	0.5310	0.1200

Table 4: Component ablation of the shared access parent and centered route field. Rows beginning with “+” add one component to the frozen acquired checkpoint, and Affirm-Direct combines both components. PopQA-20, PopQA-60, and TOFU report RL and EM on their locked test splits. All learned operators edit decoder block 20 and use development-matched edit norms.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Acquisition	Operator train	Development	Test
Balanced PopQA	1,000	600	199	201
PopQA-20	14,267	2,853	2,848	2,847
PopQA-60	14,267	8,560	2,848	2,847
TOFU	4,000	2,400	800	800
SLUE-SQA-5 transfer	–	0	0	256

Appendix

Table 5: Dataset and split accounting. Acquisition is the text-update population seen by the acquired checkpoint.

Stage or method	Trainable object	Epochs	Learning rate	Seed
Paired geometry	$D$ and temporary readout	1,000 steps	0.001	5907
Affirm-Direct Stage A	Shared vector $g$	3	0.01	8307
Affirm-Direct Stage B	Local readout $U$	1	0.0001	6007
Affirm-Expert Stage A	Router, 64 experts, and scale	3	0.001/0.01/0.0001	8307
Affirm-Expert Stage B	Local readout $U$	1	0.0001	6007
Affirm-K1 Stage A	Direction $e$ and scalar $s$	3	0.01/0.0001	8307

Appendix

Table 6: Main operator optimization. Only the listed objects are trainable, and the acquired checkpoint remains frozen.

Method	Training data	Mechanism and inference inputs
DiReFT ( Wu et al., 2024 )	600 balanced, 2,853 PopQA-20, or 2,400 TOFU rows	Rank-64 low-rank block-20 edit with a development-selected source dose
LoReFT ( Wu et al., 2024 )	600 balanced, 2,853 PopQA-20, or 2,400 TOFU rows	Rank-64 orthogonal low-rank block-20 edit with a development-selected source dose
L2S ( Parekh et al., 2025 )	600 balanced, 2,853 PopQA-20, or 2,400 TOFU paired states	Context-conditioned predictor with a 100-dimensional bottleneck, with intervention strength calibrated by matching edit norms on source states without answer scores
SALSA ( Yegorova et al., 2026 )	600 balanced, 2,853 PopQA-20, or 2,400 TOFU rows	One norm-preserving steering activation vector with a development-selected or source-state norm-matched dose
Angle ( Xiang et al., 2025 )	Paired speech and text development states only	No fitted operator, a paired text-question donor, and the development-selected all-coordinate angle rule
Length ( Xiang et al., 2025 )	Paired speech and text development states only	No fitted operator, a paired text-question donor, and the development-selected bottom-three length rule

Appendix

Table 7: Baseline training data and inference inputs. Balanced PopQA, full PopQA-20, and TOFU use separately fitted artifacts. Angle and Length are donor-assisted controls because they receive the paired text question at inference.

Comparator	$\Delta$ RL $\pm$ SE	$\Delta$ EM $\pm$ SE
DiReFT	+0.03731 $\pm$ 0.02074	+0.03980 $\pm$ 0.02122
LoReFT	+0.03234 $\pm$ 0.02048	+0.02985 $\pm$ 0.02120
SALSA	+0.01990 $\pm$ 0.01604	+0.01990 $\pm$ 0.01719
Expert Parent	−0.00249 $\pm$ 0.01229	−0.00498 $\pm$ 0.01325

Appendix

Table 8: Affirm-Direct minus matched single-state comparators on balanced PopQA. We estimate SEs from 10,000 bootstrap resamples of duplicate-question groups.

Method	$\Delta$ Span $\pm$ SE	$\Delta$ RL $\pm$ SE	$\Delta$ EM $\pm$ SE
DiReFT	+0.05469 $\pm$ 0.01720	+0.05456 $\pm$ 0.01737	+0.03906 $\pm$ 0.01388
LoReFT	+0.07031 $\pm$ 0.02386	+0.07734 $\pm$ 0.02225	+0.07031 $\pm$ 0.01610
L2S	−0.01953 $\pm$ 0.02197	−0.01758 $\pm$ 0.02083	+0.02344 $\pm$ 0.01828
SALSA	−0.08984 $\pm$ 0.03081	−0.07070 $\pm$ 0.02643	+0.02344 $\pm$ 0.02400
Affirm-Direct	+0.08594 $\pm$ 0.02507	+0.09408 $\pm$ 0.02450	+0.06641 $\pm$ 0.01944

Appendix

Table 9: No-refit SLUE method minus its paired No Repair replay. We estimate SEs from 10,000 bootstrap resamples of the 50 speakers.

Setting	Comparator	$\Delta$ RL $\pm$ SE	$\Delta$ EM $\pm$ SE
PopQA-20	Direct Parent	+0.01974 $\pm$ 0.00425	+0.01510 $\pm$ 0.00442
	Route Field	+0.01412 $\pm$ 0.00484	+0.00983 $\pm$ 0.00495
	Expert Parent	+0.00083 $\pm$ 0.00363	−0.00176 $\pm$ 0.00381
PopQA-60	Direct Parent	+0.02232 $\pm$ 0.00494	+0.01826 $\pm$ 0.00511
	Route Field	+0.01135 $\pm$ 0.00507	+0.00667 $\pm$ 0.00521
	Expert Parent	+0.00226 $\pm$ 0.00494	+0.01054 $\pm$ 0.00600

Appendix

Table 10: Affirm-Direct minus full-scale component and parent comparators. SEs use the group-bootstrap procedures in Appendix A.5 .

Comparator	$\Delta$ Span $\pm$ SE	$\Delta$ RL $\pm$ SE	$\Delta$ EM $\pm$ SE
Expert Parent (block 20)	+0.0430 $\pm$ 0.0255	+0.0218 $\pm$ 0.0216	+0.1094 $\pm$ 0.0251
Affirm-Expert (block 4)	+0.0195 $\pm$ 0.0282	+0.0060 $\pm$ 0.0266	+0.0430 $\pm$ 0.0185

Appendix

Table 11: Block-4 Affirm-Direct minus its comparators on no-refit SLUE. We estimate SEs from 10,000 bootstrap resamples of the 50 speakers. The expert-parent row uses a different intervention block.

		PopQA-20		PopQA-60
Method	Block	RL	EM	RL	EM
No Repair	–	0.1787	0.1500	0.1787	0.1500
Direct Parent	20	0.3156	0.2986	0.3269	0.3087
Route Field	20	0.3212	0.3038	0.3379	0.3203
Expert Parent	20	0.3345	0.3154	0.3470	0.3165
Affirm-Direct	20	0.3353	0.3137	0.3493	0.3270

Appendix

Table 12: Full-scale PopQA factual access. PopQA-20 and PopQA-60 use operator-training budgets of approximately 20% and 60% of the full acquisition population and share the same locked test set. Block is shown because only same-block rows are controlled architectural comparisons.

Method and setting	Development RL	Development EM	Mean edit norm
Affirm-Direct , PopQA-20	0.3462 $\pm$ 0.0018	0.3246 $\pm$ 0.0017	25.469 $\pm$ 0.020
Affirm-Expert , TOFU	0.5619 $\pm$ 0.0019	0.1571 $\pm$ 0.0031	$\approx$ 19.232

Appendix

Table 13: Stage-B stability across three training seeds. RL and EM are reported as mean

\pm

sample standard deviation.

			No Repair		Parent		Full field
Backbone	Split	Block	RL	EM	RL	EM	RL	EM
Qwen2.5-Omni-7B	Development	4	0.4695	0.0713	0.5575	0.1613	0.5585	0.1563
Kimi-Audio	Development	20	0.5655	0.0875	0.5973	0.1338	0.5960	0.1350
Kimi-Audio	Test	20	0.6321	0.0850	–	–	0.6451	0.1188
MiMo-Audio	Development	25 $\to$ 36	0.7175	0.2450	0.7113	0.2563	0.7130	0.2575

Appendix

Table 14: Affirm-K1 characterization across speech-language backbones on TOFU. Each reported score is computed over 800 questions. Development rows show No Repair, the parent, and the full field for each backbone. For Kimi, we also report test results after model selection on development data.

Method	Learned values	RL	EM
No Repair	0	0.2736	0.2189
Mean route shift	3,584	0.5323	0.5224
Representation-only route map	458,752	0.5808	0.5672
Residual KRR ( Saunders et al., 1998 )	4,300,800	0.5808	0.5672
Affirm-Direct	462,336	0.6318	0.6219

Appendix

Table 15: Representation controls on the 201-item balanced PopQA test. The mean edit norms are 22.51 for the controls and 22.62 for Affirm-Direct .

Method	Unique pairs	Selected epoch	RL	EM	Time (s)	Peak GiB	Learned values
Route map	75	–	0.5343	0.5226	123.0	21.31	0.46M
Speech LoRA	75	3	0.5419	0.5276	267.4	17.61	40.37M
Route map	150	–	0.5720	0.5628	202.6	21.42	0.46M
Speech LoRA	150	2	0.5972	0.5829	339.9	17.69	40.37M
Route map	300	–	0.5494	0.5327	468.7	21.42	0.46M
Speech LoRA	300	3	0.6323	0.6231	575.5	17.69	40.37M

Appendix

Table 16: Matched-data development comparison of a decoder-trained route map and Speech LoRA. The route map uses the empirical mean route difference in place of Affirm ’s trained access parent and the unmodified speech output as its preservation target. Its learned-value count includes both projection matrices and the empirical mean route difference. Time is standalone end-to-end training time in seconds and excludes development generation. Speech LoRA checkpoint times are reconstructed from checkpoint timestamps and stored full-run duration, so they are approximate.

Setting	Method	Pairs	Trainable	Deployed change	RL	EM
Balanced PopQA	No Repair	0	0	None	0.2736	0.2189
	Affirm-Direct	600	0.46M	One-state external operator	0.6318	0.6219
	Speech LoRA	600	40.37M	Internal adapters in 28 blocks	0.6443	0.6368
	ASR cascade	0	0	External ASR and a second text pass	0.6990	0.6915
	Full speech supervision	1,000	Full model	Full-model update	0.9801	0.9801
PopQA-60	No Repair	0	0	None	0.1787	0.1500

Appendix

Table 17: Accuracy and mutability across frozen and relaxed deployment contracts. All scores are computed on the full test splits. Speech LoRA and S2T-KD insert rank-16 adapters throughout all 28 decoder blocks, and S2T-KD adds a temperature-2 forward-KL term with weight 0.5 toward the text-route output distribution. The ASR cascade transcribes each spoken question with Whisper-large-v3 and answers the transcript with the acquired checkpoint.

Multimodal Knowledge Editing

See all 10 papers

Localizing and Editing Knowledge in Large Audio-Language Models
Mar 15, 2026Sung Kyun Chung, Jiaheng Dong, Qiuchi Hu +3Concept Localization in Language Models Audio-Language Models
The University of Melbourne, Australia · Wuhan University, China · University of Auckland, New Zealand
Efficiently Adapting Spoken Language Models for the Singaporean Context
Jul 11, 2026Ng Jia Sheng JasonSpoken Language Understanding Multi-Task Learning
Language AI R&D, xData Home Team Science & Technology Agency (HTX), Singapore
Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions
Oct 12, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Language Models Language Model Distillation
TTI-Chicago · Stony Brook University

Mar 15, 2026J/K move · Enter open · S save

Localizing and Editing Knowledge in Large Audio-Language Models

Sung Kyun Chung, Jiaheng Dong, Qiuchi Hu, Gongping Huang+2

The University of Melbourne, Australia · Wuhan University, China · University of Auckland, New Zealand

Concept Localization in Language Models Audio-Language Models Language Model Editing Factual Knowledge in Language Models

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have shown strong performance in speech understanding, making speech a natural interface for accessing factual information. Yet they are trained on static corpora and may encode incorrect facts. Existing model editing methods localize and update facts in text-only LLMs, but do not account for continuous speech representations, or where knowledge is stored across acoustic or language modules, or their cross-modal module. We construct the first audio benchmark for knowledge localization and editing in LALMs and propose a speech-driven locate-then-edit framework. First, we use speech-aware causal tracing to localize layers and modules that support factual retrieval and then apply editing at identified sites. Experiments show that factual knowledge is jointly encoded in audio and text modules, and that audio editing yields more effective updates than text editing or fine-tuning, enabling fine-grained knowledge control in speech AI systems.

Abstract

Multimodal Knowledge Editing

Explore similar work

Localizing and Editing Knowledge in Large Audio-Language Models

Efficiently Adapting Spoken Language Models for the Singaporean Context

Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions