Low-Rank Frequency Convolution and Noise-Range Augmentation for Real-Time Pitch Estimation on Edge Devices

Venkat Suprabath Bitra; Homayoon Beigi

eess.ASSep 20, 2026

Low-Rank Frequency Convolution and Noise-Range Augmentation for Real-Time Pitch Estimation on Edge Devices

Authors: Venkat Suprabath Bitra, Homayoon Beigi

Organizations: Nonlinear Control Lab, Columbia University, New York, USA · Dept. of Electrical Engineering, Columbia University, New York, USA · Dept. of Mechanical Engineering, Columbia University, New York, USA · Recognition Technologies, Inc., New York, USA

Abstract

Pitch estimation on an edge device is constrained in three ways at once. The model must be small, it must stay accurate when the input is noisy, and one frame must be produced inside the frame period. In this report the Frequency Convolution Network (FrCN) of our earlier work is factored into a low-rank form. The number of parameters is reduced by 35.9%, from 17,787 to 11,397, and accuracy is not reduced, either in domain or on two corpora the model was never trained on. The range of the noise used during training is also shown to dominate the architecture in setting how the model behaves when the interference is severe. When the training noise floor is lowered from +6.02 dB to -20 dB, 0.35 points of clean RPA50 are lost and RPA50 at -20 dB is raised from 2.44 to 22.41. This effect is about two orders of magnitude larger than any architectural effect that was measured. The semi-orthogonal constraint used in TDNN-F is found to be redundant with the normalization inside the bottleneck, and accuracy is reduced when both are applied. Whether the factorization saves time depends on the runtime: in eager PyTorch the factored model is 45% slower, while in a compiled kernel it is 15% faster. For deployment, a small C kernel was written. It needs 83 kB on disk and no runtime library beyond libc and libm. It is faster than OpenBLAS on all four CPUs that were tested, faster than ONNX Runtime by 3.8 times, and faster than PyTorch by 13 times. Its output was checked against PyTorch on 271,893 held-out frames per model, and the same pitch bin was selected on every one of them.

Figures & tables

$\beta_{\max}$	Training noise floor (dB)
$0.5$	$+6.02$
$1.0$	$0.00$
$2.0$	$-6.02$
$4.0$	$-12.04$
$10.0$	$-20.00$

Table 1: The five augmentation settings. The floor is the worst signal-to-noise ratio that can be drawn during training.

Set	Model	Par.	RPA50	RCA50	MnE	MdE
MDB	FrCN	17.8k	98.98	99.15	0.078	0.019
MDB	Low-rank	11.4k	99.07	99.22	0.071	0.019
PTDB	FrCN	17.8k	88.76	91.48	0.649	0.082
PTDB	Low-rank	11.4k	88.62	91.45	0.663	0.084
MIR-1K	FrCN	17.8k	95.23	95.37	0.263	0.085
MIR-1K	Low-rank	11.4k	95.25	95.38	0.273	0.086

Table 2: Clean in-domain accuracy, mean over three seeds. MnE and MdE are in semitones.

Target	Model	RPA50	$\Delta$
MIR-1K	FrCN	93.02 (0.05)	–
MIR-1K	Low-rank	93.08 (0.08)	+0.06
PTDB	FrCN	85.36 (0.13)	–
PTDB	Low-rank	85.37 (0.05)	+0.01

Table 3: Cross-corpus accuracy. Both models are trained on MDB and evaluated with no retraining. RPA50, mean over three seeds, seed standard deviation in parentheses.

\Delta

is the paired per-seed difference from FrCN.

Constraint and normalization
Configuration	Par.	RPA50
Norm, no constraint (proposed)	11.4k	99.07 $\pm$ 0.00
Norm $+$ semi-orthogonal	11.4k	98.90 $\pm$ 0.05
No norm, no constraint	11.3k	98.96 $\pm$ 0.13
No norm $+$ semi-orthogonal	11.3k	98.92 $\pm$ 0.01
Cost of the constraint, paired by seed

Table 4: Ablations on MDB held out, RPA50 over three seeds. The semi-orthogonal constraint and the bottleneck normalization regulate the same quantity and do not compose.

Floor	Model	Clean	10 dB	0 dB	$-6$ dB	$-12$ dB	$-20$ dB
$+6.02$ dB	FrCN	98.98	94.94	60.23	22.50	6.40	1.93
$+6.02$ dB	Low-rank	99.07	94.90	62.39	25.44	7.34	2.44
$0.00$ dB	FrCN	98.88	94.50	72.66	21.24	3.81	1.55
$0.00$ dB	Low-rank	98.94	94.48	73.89	27.45	7.31	2.40
$-6.02$ dB	FrCN	98.71	93.80	73.51	50.85	14.62	2.77
$-6.02$ dB	Low-rank	98.86	93.74	74.02	52.13	19.50	4.40

Table 5: Training noise floor against evaluation SNR. MDB held out, RPA50, mean over three seeds. Best per column in bold.

Model	Policy	Width	Size (kB)	RPA50	$\Delta$
FrCN	nf	32	69.5	98.977 (0.010)	–
		16	35.6	98.977 (0.011)	+0.000
		8	18.6	98.980 (0.009)	+0.003
		4	10.1	97.619 (1.040)	-1.358
	nq	32	69.5	98.977 (0.010)	–
		16	35.2	98.976 (0.010)	-0.000

Table 6: Effect of weight width on accuracy and on size. RPA50 on MDB held out, mean over three seeds, seed standard deviation in parentheses.

Model	Par.	MACs	PyTorch	Naive C	Ours
FrCN	17.8k	4.42M	535	175.1	68.0
Low-rank $M=9$	11.4k	2.67M	777	125.0	58.0
no inner norm	11.3k	2.67M	666	122.9	55.1
Low-rank $M=6$	8.1k	1.80M	751	93.2	52.1
Low-rank $M=12$	14.7k	3.55M	779	159.5	75.9

Table 7: CPU cost of one frame, batch size one, single thread, idle machine. The two runtimes rank the models in opposite orders. The times for PyTorch, Naive C and Ours are in

\mu

Inference path	FrCN		Low-rank $M=9$
Inference path	$\mu$ s	vs ours	$\mu$ s	vs ours
PyTorch eager, fp32	535	7.9 $\times$	777	13.4 $\times$
ONNX Runtime, fp32	150	2.2 $\times$	223	3.8 $\times$
ONNX Runtime, int8	1285	18.9 $\times$	1409	24.3 $\times$
Our C, naive kernel	175	2.6 $\times$	125	2.2 $\times$
Our C, im2col + OpenBLAS	86	1.3 $\times$	76	1.3 $\times$

Table 8: Latency of one frame on the same host, batch size one, single thread. Each model has its own ratio against our kernel.

Path	$\mu$ s/frame	Deploy (MB)	Peak RSS (MB)	Init (ms)
Our C, register blocked	58	0.08	2.4	0
Our C, naive kernel	125	0.08	2.4	0
Our C with OpenBLAS	76	45	5.2	0
ONNX Runtime	223	65	67	14
PyTorch eager	777	686	226	514

Table 9: What each inference path costs beyond its latency, low-rank model. PyTorch is its CPU-only build.

Model	Kernel	x86-64	Pi 4	Pi 5	Orin NX
Model	Kernel	4949 MHz	1800 MHz	2400 MHz	1984 MHz
FrCN	OpenBLAS	87	1961	788	800
	ours	68	1330	449	501
	RTF	0.007	0.137	0.046	0.051
Low-rank $M=9$	OpenBLAS	77	1875	768	758
	ours	58	1197	414	438

Table 10: Our kernel on four CPUs, one frame, single thread, clock pinned, fp32 compute. RTF is the fraction of the

10

ms hop, at int8 storage. The timings for OpenBLAS and ours are in

\mu

Model	Width	Flash	x86-64	Pi 4	Pi 5	Orin NX
Model	Width	(kB)	( $\mu$ s)	( $\mu$ s)	( $\mu$ s)	( $\mu$ s)
FrCN	fp32	67.1	68	1330	449	501
	fp16	34.4	84	1417	487	536
	int16	35.2	69	1366	455	505
	int8	18.8	69	1374	457	506
	int4	10.7	79	1476	480	527

Table 11: Every stored width on every device, our kernel, one frame, single thread. The row marked int8 arith. computes in integer, every other row computes in float.

Edge Inference

Edge inference runs trained AI models on nearby devices such as phones, cameras, robots, and embedded computers under tight resource and latency limits.

See all 228 papers

Pitch Smoothing Using Relative Interval Networks
Sep 30, 2026Chin-Yun Yu, Chi-Jen Peng, Li Su +1Music Information Retrieval
Centre for Digital Music, Queen Mary University of London, UK · Department of Information Management, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan
Radio-Frequency Convolutional Neural Networks
Sep 16, 2026Zhihui Gao, Shi-Yuan Ma, Yiran Chen +2Efficient Neural Network Inference Edge Inference
Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA. · Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs
Jul 28, 2026Gyeongmin KimEfficient Neural Network Inference Speech Enhancement
Department of Computer Science, Hanyang University, Seoul, Korea

Sep 30, 2026J/K move · Enter open · S save

Pitch Smoothing Using Relative Interval Networks

Chin-Yun Yu, Chi-Jen Peng, Li Su, György Fazekas

Centre for Digital Music, Queen Mary University of London, UK · Department of Information Management, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan

Music Information Retrieval

Open paper AskPDF arXiv

Pitch tracking systems typically couple a per-frame fundamental frequency (

F_0

) estimator with a temporal smoothing stage to obtain continuous trajectories. Conventional Viterbi smoothers enforce first-order continuity but lack long-term temporal awareness and could lock into octave errors across corrupted frames. We propose Relative Interval Networks (RIN), a trajectory smoothing framework that reconciles per-frame pitch estimates with data-driven multi-hop pitch differences. We extract robust relative pitch intervals across arbitrary frame offsets using Variable-Q Transform cross-correlation. We formulate pitch smoothing as an

L_1

-norm optimization problem and prove its equivalence to a minimum cost circulation problem, solved efficiently via linear programming. Evaluations across speech, singing, and instrumental datasets show that RIN substantially improves weak estimators, matches or outperforms Viterbi decoding at a comparable computational cost, and provides superior robustness under certain acoustic degradation.

Abstract

Edge Inference

Explore similar work

Pitch Smoothing Using Relative Interval Networks

Radio-Frequency Convolutional Neural Networks

faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs