Low-Rank Frequency Convolution and Noise-Range Augmentation for Real-Time Pitch Estimation on Edge Devices

Venkat Suprabath Bitra; Homayoon Beigi

eess.ASSep 20, 2026

Low-Rank Frequency Convolution and Noise-Range Augmentation for Real-Time Pitch Estimation on Edge Devices

Authors: Venkat Suprabath Bitra, Homayoon Beigi

Organizations: Nonlinear Control Lab, Columbia University, New York, USA · Dept. of Electrical Engineering, Columbia University, New York, USA · Dept. of Mechanical Engineering, Columbia University, New York, USA · Recognition Technologies, Inc., New York, USA

Abstract

Pitch estimation on an edge device is constrained in three ways at once. The model must be small, it must stay accurate when the input is noisy, and one frame must be produced inside the frame period. In this report the Frequency Convolution Network (FrCN) of our earlier work is factored into a low-rank form. The number of parameters is reduced by 35.9%, from 17,787 to 11,397, and accuracy is not reduced, either in domain or on two corpora the model was never trained on. The range of the noise used during training is also shown to dominate the architecture in setting how the model behaves when the interference is severe. When the training noise floor is lowered from +6.02 dB to -20 dB, 0.35 points of clean RPA50 are lost and RPA50 at -20 dB is raised from 2.44 to 22.41. This effect is about two orders of magnitude larger than any architectural effect that was measured. The semi-orthogonal constraint used in TDNN-F is found to be redundant with the normalization inside the bottleneck, and accuracy is reduced when both are applied. Whether the factorization saves time depends on the runtime: in eager PyTorch the factored model is 45% slower, while in a compiled kernel it is 15% faster. For deployment, a small C kernel was written. It needs 83 kB on disk and no runtime library beyond libc and libm. It is faster than OpenBLAS on all four CPUs that were tested, faster than ONNX Runtime by 3.8 times, and faster than PyTorch by 13 times. Its output was checked against PyTorch on 271,893 held-out frames per model, and the same pitch bin was selected on every one of them.

Figures & tables

$\beta_{\max}$	Training noise floor (dB)
$0.5$	$+6.02$
$1.0$	$0.00$
$2.0$	$-6.02$
$4.0$	$-12.04$
$10.0$	$-20.00$

Table 1: The five augmentation settings. The floor is the worst signal-to-noise ratio that can be drawn during training.

Set	Model	Par.	RPA50	RCA50	MnE	MdE
MDB	FrCN	17.8k	98.98	99.15	0.078	0.019
MDB	Low-rank	11.4k	99.07	99.22	0.071	0.019
PTDB	FrCN	17.8k	88.76	91.48	0.649	0.082
PTDB	Low-rank	11.4k	88.62	91.45	0.663	0.084
MIR-1K	FrCN	17.8k	95.23	95.37	0.263	0.085
MIR-1K	Low-rank	11.4k	95.25	95.38	0.273	0.086

Table 2: Clean in-domain accuracy, mean over three seeds. MnE and MdE are in semitones.

Target	Model	RPA50	$\Delta$
MIR-1K	FrCN	93.02 (0.05)	–
MIR-1K	Low-rank	93.08 (0.08)	+0.06
PTDB	FrCN	85.36 (0.13)	–
PTDB	Low-rank	85.37 (0.05)	+0.01

Table 3: Cross-corpus accuracy. Both models are trained on MDB and evaluated with no retraining. RPA50, mean over three seeds, seed standard deviation in parentheses.

\Delta

is the paired per-seed difference from FrCN.

Constraint and normalization
Configuration	Par.	RPA50
Norm, no constraint (proposed)	11.4k	99.07 $\pm$ 0.00
Norm $+$ semi-orthogonal	11.4k	98.90 $\pm$ 0.05
No norm, no constraint	11.3k	98.96 $\pm$ 0.13
No norm $+$ semi-orthogonal	11.3k	98.92 $\pm$ 0.01
Cost of the constraint, paired by seed

Table 4: Ablations on MDB held out, RPA50 over three seeds. The semi-orthogonal constraint and the bottleneck normalization regulate the same quantity and do not compose.

Floor	Model	Clean	10 dB	0 dB	$-6$ dB	$-12$ dB	$-20$ dB
$+6.02$ dB	FrCN	98.98	94.94	60.23	22.50	6.40	1.93
$+6.02$ dB	Low-rank	99.07	94.90	62.39	25.44	7.34	2.44
$0.00$ dB	FrCN	98.88	94.50	72.66	21.24	3.81	1.55
$0.00$ dB	Low-rank	98.94	94.48	73.89	27.45	7.31	2.40
$-6.02$ dB	FrCN	98.71	93.80	73.51	50.85	14.62	2.77
$-6.02$ dB	Low-rank	98.86	93.74	74.02	52.13	19.50	4.40

Table 5: Training noise floor against evaluation SNR. MDB held out, RPA50, mean over three seeds. Best per column in bold.

Model	Policy	Width	Size (kB)	RPA50	$\Delta$
FrCN	nf	32	69.5	98.977 (0.010)	–
		16	35.6	98.977 (0.011)	+0.000
		8	18.6	98.980 (0.009)	+0.003
		4	10.1	97.619 (1.040)	-1.358
	nq	32	69.5	98.977 (0.010)	–
		16	35.2	98.976 (0.010)	-0.000

Table 6: Effect of weight width on accuracy and on size. RPA50 on MDB held out, mean over three seeds, seed standard deviation in parentheses.

Model	Par.	MACs	PyTorch	Naive C	Ours
FrCN	17.8k	4.42M	535	175.1	68.0
Low-rank $M=9$	11.4k	2.67M	777	125.0	58.0
no inner norm	11.3k	2.67M	666	122.9	55.1
Low-rank $M=6$	8.1k	1.80M	751	93.2	52.1
Low-rank $M=12$	14.7k	3.55M	779	159.5	75.9

Table 7: CPU cost of one frame, batch size one, single thread, idle machine. The two runtimes rank the models in opposite orders. The times for PyTorch, Naive C and Ours are in

\mu

Inference path	FrCN		Low-rank $M=9$
Inference path	$\mu$ s	vs ours	$\mu$ s	vs ours
PyTorch eager, fp32	535	7.9 $\times$	777	13.4 $\times$
ONNX Runtime, fp32	150	2.2 $\times$	223	3.8 $\times$
ONNX Runtime, int8	1285	18.9 $\times$	1409	24.3 $\times$
Our C, naive kernel	175	2.6 $\times$	125	2.2 $\times$
Our C, im2col + OpenBLAS	86	1.3 $\times$	76	1.3 $\times$

Table 8: Latency of one frame on the same host, batch size one, single thread. Each model has its own ratio against our kernel.

Path	$\mu$ s/frame	Deploy (MB)	Peak RSS (MB)	Init (ms)
Our C, register blocked	58	0.08	2.4	0
Our C, naive kernel	125	0.08	2.4	0
Our C with OpenBLAS	76	45	5.2	0
ONNX Runtime	223	65	67	14
PyTorch eager	777	686	226	514

Table 9: What each inference path costs beyond its latency, low-rank model. PyTorch is its CPU-only build.

Model	Kernel	x86-64	Pi 4	Pi 5	Orin NX
Model	Kernel	4949 MHz	1800 MHz	2400 MHz	1984 MHz
FrCN	OpenBLAS	87	1961	788	800
	ours	68	1330	449	501
	RTF	0.007	0.137	0.046	0.051
Low-rank $M=9$	OpenBLAS	77	1875	768	758
	ours	58	1197	414	438

Table 10: Our kernel on four CPUs, one frame, single thread, clock pinned, fp32 compute. RTF is the fraction of the

10

ms hop, at int8 storage. The timings for OpenBLAS and ours are in

\mu

Model	Width	Flash	x86-64	Pi 4	Pi 5	Orin NX
Model	Width	(kB)	( $\mu$ s)	( $\mu$ s)	( $\mu$ s)	( $\mu$ s)
FrCN	fp32	67.1	68	1330	449	501
	fp16	34.4	84	1417	487	536
	int16	35.2	69	1366	455	505
	int8	18.8	69	1374	457	506
	int4	10.7	79	1476	480	527

Table 11: Every stored width on every device, our kernel, one frame, single thread. The row marked int8 arith. computes in integer, every other row computes in float.

Edge Inference

Edge inference runs trained AI models on nearby devices such as phones, cameras, robots, and embedded computers under tight resource and latency limits.

See all 228 papers

Sep 30, 2026

Pitch Smoothing Using Relative Interval Networks

Pitch tracking systems typically couple a per-frame fundamental frequency (

F_0

) estimator with a temporal smoothing stage to obtain continuous trajectories. Conventional Viterbi smoothers enforce first-order continuity but lack long-term temporal awareness and could lock into octave errors across corrupted frames. We propose Relative Interval Networks (RIN), a trajectory smoothing framework that reconciles per-frame pitch estimates with data-driven multi-hop pitch differences. We extract robust relative pitch intervals across arbitrary frame offsets using Variable-Q Transform cross-correlation. We formulate pitch smoothing as an

L_1

-norm optimization problem and prove its equivalence to a minimum cost circulation problem, solved efficiently via linear programming. Evaluations across speech, singing, and instrumental datasets show that RIN substantially improves weak estimators, matches or outperforms Viterbi decoding at a comparable computational cost, and provides superior robustness under certain acoustic degradation.

Chin-Yun Yu, Chi-Jen Peng, Li Su +1

Centre for Digital Music, Queen Mary University of London, UK · Department of Information Management, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan

Music Information Retrieval

Ask PDF

Sep 16, 2026

Radio-Frequency Convolutional Neural Networks

Running artificial intelligence (AI) models directly on edge devices such as smartphones, wearables, and drones offers low latency, pervasive scalability, and data privacy, but these devices rarely carry the computing capability that modern neural networks demand. Edge accelerators have been developed in response, yet each adds computing hardware to devices already constrained in size, weight, power, and cost (SWaP-C). An alternative lies in what these devices already carry: the frequency mixer in every wireless radio multiplies signals in time, natively performing convolution in the frequency domain. Here we introduce radio-frequency convolutional neural networks (RF-CNNs), which repurpose existing communication hardware for CNN inference. Multi-channel convolutions are mapped onto frequency tones for a passive mixer to execute in a single pass. We experimentally demonstrate that RF-CNN runs deep CNNs up to 26.4 million parameters and nine layers from classification of wireless signals and images to controllable image generation, close to full-precision performance. Because the weights arrive over the air and the analog hardware is shared with communication, the edge device spends energy only on data preparation and readout-down to 0.72 femtojoules per multiply-accumulate, two orders of magnitude less than it would cost on an added digital processor. These results suggest that deployed wireless infrastructure can bring efficient, state-of-the-art AI inference to the billions of devices it already connects.

Zhihui Gao, Shi-Yuan Ma, Yiran Chen +2

Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA. · Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.

Efficient Neural Network Inference Edge Inference Convolutional Neural Networks

Ask PDF

Jul 28, 2026

faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs

This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.

Gyeongmin Kim

Department of Computer Science, Hanyang University, Seoul, Korea

Efficient Neural Network Inference Speech Enhancement On-Device Inference

Ask PDF

Abstract

Edge Inference

Explore similar work

Pitch Smoothing Using Relative Interval Networks

Radio-Frequency Convolutional Neural Networks

faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs