ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometry

Yuhan Liu; Yuxuan Ou; Ruoxi Su; Mohamed Ahmed Zaki; Yunbo Long

cs.SDOct 2, 2026

ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometry

Authors: Yuhan Liu, Yuxuan Ou, Ruoxi Su, Mohamed Ahmed Zaki, Yunbo Long

Organizations: Department of Engineering, University of Cambridge · Department of Engineering Science, University of Oxford · Department of Computing & Software Systems, University of Washington Bothell

Abstract

Speech delivery varies with both the requested paralinguistic attribute and the linguistic content. We introduce ParaGeo, a matched-content decomposition of paralinguistic variation in a frozen speech language model. Synthesized audio tokens are replayed with a fixed listening prompt; pooled key/value (K/V) representations are centered and projected into a shared low-dimensional space. Our GLM-4-Voice probe spans 80 requested controls from 12 benchmark families across eight sentences. With a globally fitted calibration basis, content-held-out centroid accuracy using this basis is 9.49% versus a 1.25% permutation baseline; same-label cross-content cosine similarity is 0.285 versus 0.017, and both conditional permutation tests yield p = 0.001. A separate ten-scenario, six-style probe reveals reproducible contrast directions across scenarios. Static, additive, and temporal interventions produce attribute-, layer-, and schedule-dependent response profiles. These results provide a shared coordinate representation for measuring paralinguistic structure and an empirical starting point for latent speech control. Code is available at https://github.com/yuhanlydia/ParaGeo.

Figures & tables

Figure 1: Vocal-style changes recur across contexts. Each cell measures the cross-scenario consistency of the same style-difference direction, rather than similarity between styles. Results use ten CRAD scenarios and two seeds. All 15 unique contrasts pass Benjamini–Hochberg correction at a false discovery rate of 0.05.

Figure 2: ParaGeo overview. Matched-content renditions are synthesized and replayed through the frozen speech model. Audio-position K/V pooling, within-content centering, low-rank decomposition, and a content-associated projection yield shared coordinates. Static, additive, and temporal updates reuse the same basis. The diagram and waveforms are schematic.

Figure 3: Attribute-specific responses across five direction controls. Columns show all 18 attributes, ordered by mean; S1–S5 denote the five random-direction comparison seeds. Each column reuses ten inputs and the same ParaGeo outputs. Cells report direct preference; summaries are mean, minimum, and count above 50. The pooled preference is 51.50 [45.39,57.56].

Representation	80-way accuracy	Cosine
Centered full K/V	11.69%	0.152
Projected 16-D coordinates	9.49%	0.285

Table 1: Representation measurements on 590 records. Both rows are centered within sentence. Accuracy predicts the exact requested label; cosine compares the same label across sentences.

Configuration	Pref. $\uparrow$	95% CI	$\Delta$ WER $\downarrow$
All layers ( $r=16$ )	54.17	[37.50,70.83]	$+0.0055$
w/o content proj.	43.75	[27.08,60.42]	$+0.0021$
Rank 4	58.33	[41.67,75.00]	$0.0000$
Rank 8	54.17	[37.50,70.83]	$+0.0108$
Rank 32	47.92	[31.25,64.58]	$+0.0021$
Early layers	60.42	[45.83,75.00]	$+0.0160$

Table 2: Dynamic control and text fidelity. Fixed 24-item subset; preference against prompt-only (50 = parity), nominal 95% bootstrap CIs, and absolute change in text-channel WER.

Family	Inputs	Preference
Emotion	12	54.17
Pace	12	75.00
Pitch	12	45.83
Rhythm	12	37.50
Timbre	4	62.50
Volume	8	37.50

Table 3: Dynamic family profiles. All 60 inputs at exploratory

\alpha=0.25

; preference against prompt-only.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Question	Evidence in this record	Interpretation
Does a requested change recur across content?	Six-style cross-scenario contrasts; 80-way content-held-out centroids; conditional permutations	Request-associated structure in a specified replay representation.
What does the projection retain?	Raw-SVD variance; full versus projected decoding and cosine	Increased alignment with a loss of some fine-grained label distinctions.
Where do updates respond favorably?	Complete rank/layer/schedule ablations; 18-attribute control matrix; task-family profiles	Descriptive operating settings, with nominal subset intervals and complete unfavorable entries.
Is there an aggregate control gain?	Development selection; full held-out results; direct random and full-space comparisons	The current aggregate comparisons do not establish general superiority.

Appendix

Table A1: How to read the evidence. Each row names a distinct measurement obligation. Preference uses a 0–100 scale with 50 denoting parity; differences in WER elsewhere are absolute fractions.

ID	Target text
cal:00	The morning train arrived exactly on time.
cal:01	Please place the blue notebook beside the lamp.
cal:02	We will review the final report after lunch.
cal:03	A small bird landed quietly on the garden fence.
cal:04	The museum opens at nine o’clock tomorrow.
cal:05	I left the keys on the wooden table near the door.

Appendix

Table A2: The eight fixed calibration sentences. All controls reuse this lexical bank. The identifiers match the released calibration-sentence file.

Family	Count	Requested controls
Age	4	child’s voice; youthful voice; adult voice; elderly voice.
Pitch	5	very high pitch; high pitch; medium pitch; low pitch; very low pitch.
Timbre	6	bright timbre; hoarse timbre; smooth timbre; rich timbre; gentle timbre; sweet timbre.
Pace	5	very fast pace; fast pace; medium pace; slow pace; very slow pace.
Volume	5	shouting manner; loudly volume; normal volume; quietly volume; whisper.
Pause	3	clear pause after the word ‘listen’; clear pause after the word ‘but’; clear pause after the word ‘good’.

Appendix

Table A3: Complete calibration catalog: 80 controls in 12 benchmark families. Semicolons separate entries. The original benchmark wording, including “loudly volume” and distinct stress phrasings, is preserved.

Statistic	LOCO (%)	Mean cosine
Observed	9.4915	0.28531
Null mean	1.2468	0.01703
Null median	1.1864	0.01647
Null 5th percentile	0.5085	0.00443
Null 95th percentile	2.0339	0.03117
Monte Carlo $p$	0.000999	0.000999

Appendix

Table A4: The complete projected-geometry null summary. Quantiles refer to 1,000 conditional permutations; LOCO is exact 80-way centroid accuracy in percent.

Figure A1: Prescribed coordinate schedules, not measured acoustic trajectories. The x-axis counts generation forward passes. Gradual interpolation reaches one at pass 191; the sudden switch occurs at 96; constant controls provide the start, end, and midpoint comparisons. Interleaved text/audio tokens make this index different from waveform time.

Task	$\alpha$	Items	Preference	95% CI	$\Delta$ WER	Status
Static	0.25	42	53.57	[40.48,65.48]	$+0.03335$	Fidelity gate not met
Static	0.5	42	46.43	[33.33,59.52]	$+0.00413$	Selected
Static	1.0	42	42.86	[29.76,55.95]	$-0.06605$	Eligible
Static	1.5	19/42	—	—	—	Incomplete; excluded
Composition	0.25	10	70.83	[55.00,85.83]	$+0.12000$	Fidelity gate not met
Composition	0.5	10	70.00	[47.50,90.00]	$+0.12000$	Fidelity gate not met

Appendix

Table A5: Development scale selection. Preference and nominal 95% intervals are against prompt-only.

\Delta

WER is an absolute change in emitted-text WER; the tolerance is

+0.02

Task	Candidate versus comparator	Items	Preference	95% CI	Status
Static	ParaGeo vs prompt-only	168	55.36	[49.11,61.61]	Selected, $\alpha=0.5$
Static	Random vs prompt-only	168	55.95	[49.40,62.50]	Direction control
Composition	ParaGeo vs prompt-only	38	45.39	[36.18,54.82]	Exploratory
Composition	Full-space vs prompt-only	38	48.03	[37.94,58.33]	Exploratory
Composition	ParaGeo vs full-space	38	42.98	[34.87,51.10]	Exploratory direct
Dynamic	ParaGeo vs prompt-only	60	51.67	[40.00,63.33]	Exploratory

Appendix

Table A6: Complete reported task-level preference comparisons. All entries use item-averaged scores (50 = parity). “Selected” identifies the completed development-selected static arm. Composition has no admissible development scale; the complete dynamic arm uses exploratory

\alpha=0.25

Task	Items	ParaGeo	Prompt-only	$\Delta$ WER
Static	168	0.30526	0.31254	$-0.00729$
Composition	38	0.45671	0.38789	$+0.06883$
Dynamic	60	0.26093	0.23537	$+0.02556$

Appendix

Table A7: Complete ParaGeo text-channel fidelity. WER and paired changes are absolute fractions. A change of

+0.02

is two percentage points, rather than a two-percent relative increase.

Configuration	$n$	Pref.	95% CI	$\Delta$ WER
Main (all; rank 16)	24	60.42	[45.83,72.92]	$-0.0528$
No content projection	24	56.25	[39.58,70.83]	$-0.0528$
Rank 4	23	52.17	[36.96,65.22]	$-0.0403$
Rank 8	24	58.33	[41.67,75.00]	$-0.0528$
Rank 32	24	56.25	[37.50,72.92]	$+0.0065$
Early layers	24	52.08	[37.50,66.67]	$-0.0528$

Appendix

Table A8: Complete static configuration results. The first 24 eligible held-out inputs form the fixed diagnostic subset. Preference is against prompt-only (50 = parity); intervals are nominal 95% item-bootstrap CIs.

\Delta

WER is an absolute text-channel change. Static rank-4 preference has 23 judged inputs; its WER summary has 24.

Configuration	$n$	Pref.	95% CI	$\Delta$ WER
Main (all; rank 16)	24	66.67	[54.17,78.12]	$+0.1250$
No content projection	24	48.96	[35.42,61.46]	$-0.0333$
Rank 4	24	53.12	[37.50,68.75]	$-0.0333$
Rank 8	24	50.00	[38.54,61.46]	$-0.0333$
Rank 32	24	56.25	[43.75,68.75]	$+0.1125$
Early layers	24	42.71	[30.21,55.21]	$-0.0542$

Appendix

Table A9: Complete composition configuration results. The first 24 eligible held-out inputs form the fixed diagnostic subset. Preference is against prompt-only (50 = parity); intervals are nominal 95% item-bootstrap CIs.

\Delta

WER is an absolute text-channel change.

Configuration	$n$	Pref.	95% CI	$\Delta$ WER
Main (all; rank 16)	24	54.17	[37.50,70.83]	$+0.0055$
No content projection	24	43.75	[27.08,60.42]	$+0.0021$
Rank 4	24	58.33	[41.67,75.00]	$+0.0000$
Rank 8	24	54.17	[37.50,70.83]	$+0.0108$
Rank 32	24	47.92	[31.25,64.58]	$+0.0021$
Early layers	24	60.42	[45.83,75.00]	$+0.0160$

Appendix

Table A10: Complete dynamic configuration results. The first 24 eligible held-out inputs form the fixed diagnostic subset. Preference is against prompt-only (50 = parity); intervals are nominal 95% item-bootstrap CIs.

\Delta

WER is an absolute text-channel change.

Figure A2: The layer response depends on the task. All four sampled-layer configurations are shown for all three 24-input diagnostic tasks. Cells are preference estimates against prompt-only; their recorded nominal intervals appear in Tables A8 – A10 . Middle-layer injection has the largest static and dynamic estimates; all-layer composition has the largest composition estimate and a substantial text-fidelity cost (Table A9 ). The matrix is a response profile, not a corrected test of pairwise layer superiority.

Attribute	S1	S2	S3	S4	S5	Mean	Min.	$>50$
Cold	35	55	10	50	55	41	10	2/5
Contemptuous	35	30	35	50	20	34	20	0/5
Enthusiastic	55	40	50	70	80	59	40	3/5
Perfunctory	40	30	45	30	45	38	30	0/5
Polite	60	75	55	55	75	64	55	5/5
Rude	40	45	45	60	65	51	40	2/5

Appendix

Table A11: All 18 attributes against all five random-direction conditions. Each row uses ten unique inputs, with the same ParaGeo outputs reused across S1–S5. The final columns are descriptive mean, minimum, and count strictly above 50. Labels retain the source order; no unfavorable row is excluded.

Control	Pref.	95% CI	W/T/L
S1	49.44	[41.11,58.06]	75/28/77
S2	53.06	[43.61,62.22]	81/29/70
S3	49.17	[38.89,59.17]	79/19/82
S4	50.56	[40.56,60.28]	74/34/72
S5	55.28	[45.56,64.72]	85/29/66

Appendix

Table A12: Direct ParaGeo preference for each direction-control condition. Every row reuses the same 180 inputs. The source intervals and strict win/tie/loss counts are preserved. Control orientations do not provide independent ParaGeo runs.

Comparison	Preference	95% CI	Text WER (%)	$\Delta$ WER (pp)
Prompt-only	—	—	38.79	0.00
Projected sum vs prompt-only	45.39	[36.18,54.82]	45.67	$+6.88$
Full-space sum vs prompt-only	48.03	[37.94,58.33]	50.28	$+11.49$
Projected vs full-space	42.98	[34.87,51.10]	—	$-4.61$

Appendix

Table A13: Composition control and fidelity on the same 38 inputs. Preference is against prompt-only; both interventions use exploratory

\alpha=0.25

. The final row is the direct preference for projection over full-space addition. WER is in percent here, for direct comparison with the percentage-point changes.

Task	Family	Inputs	ParaGeo	Random control
Static	Age	9	66.67	50.00
Static	Attitude	22	40.91	56.82
Static	Cognitive State	22	45.45	63.64
Static	Emotion	17	64.71	64.71
Static	Non-Linguistic Vocalizations	19	63.16	63.16
Static	Pace	12	50.00	58.33

Appendix

Table A14: Every reported static and dynamic family. Both preference columns compare separately with prompt-only (50 = parity); they are not a head-to-head comparison. Static uses selected

\alpha=0.5

and dynamic uses exploratory

\alpha=0.25

. Family-specific intervals are unavailable.

Setting	Value
Backbone; weights	GLM-4-Voice-9B; int4/NF4
Temperature; top- $p$	0.8; 0.8
Maximum new tokens; waveform rate	768; 22.05 kHz
Selected layers	16, 20, 24, 28, 32, 36
Default rank; evaluated alternatives	16; 4, 8, 32
Content-basis setting	8 returned columns; rank boundary in Appendix B

Appendix

Table A15: Recorded execution settings. Restart equivalence also requires traversal and artifact identity.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding
Sep 19, 2026Nishit Anand, Jiaqi Su, Ke Chen +5Audio Understanding Audio-Language Models
Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation
Apr 22, 2026Ruohan Liu, Shukang Yin, Tao Wang +6Audio-Language Model Evaluation LLM Evaluation
1Nanjing University · 2Xiaomi

Sep 19, 2026J/K move · Enter open · S save

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

Nishit Anand, Jiaqi Su, Ke Chen, Yunyun Wang+4

Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA

Audio Understanding Audio-Language Models Audio Reasoning Audio-Language Model Evaluation

Open paper AskPDF arXiv

Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.

Abstract

Representation Geometry in Language Models

Explore similar work

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation