Learning Gradients of Homogeneous Functions

Mayur S. Gowda; Srinivasa Pranav; Shreyas Chaudhari; José M. F. Moura

cs.LGNov 1, 2026

Learning Gradients of Homogeneous Functions

Authors: Mayur S. Gowda, Srinivasa Pranav, Shreyas Chaudhari, José M. F. Moura

Organizations: Department of Electrical and Computer Engineering, Carnegie Mellon University

Abstract

Homogeneous gradient functions have widespread significance, including applications in maximum energy subspace selection and retrieval problems. Euler's homogeneous function theorem specifies how the potential associated with a positively k-homogeneous gradient field can be recovered directly from the field itself. We use Euler's theorem to build on GradNets and propose a homogeneous gradient network architecture that is guaranteed to correspond to the gradient of a homogeneous function. Then, we describe how monotone homogeneous GradNets universally approximate the gradients of k-homogeneous convex functions. The inductive bias of homogeneous GradNets enables them to outperform baseline ICNNs by up to 7dB in our max energy subspace projection experiment and by up to 13% higher accuracy, 98% fewer parameters, and 70% fewer FLOPS in our amortized inner product search experiment.

Figures & tables

Figure 1: Ground-truth and learned potentials on a 2-D orthonormal slice of a

d=16

maximum energy subspace problem. White lines indicate boundaries where the maximizing projector changes.

Figure 2: $d=16$ maximum energy subspace problem. Left and center: true vs. predicted potentials for ICNN and GradNet; dashed line denotes exact agreement. Right: absolute potential error distributions.

$d$	16	32	64	128	256
ICNN	-5.04	-4.36	-4.13	-3.99	-3.89
GradNet	-11.97	-9.74	-7.66	-5.63	-4.89

Table 1: Maximum projected subspace energy task test gradient MSE (dB) across ambient dimensions

d

. Lower is better.

Model	Acc. $\uparrow$	MRR $\uparrow$	[email protected]% $\uparrow$	Params $\downarrow$	Proj. FLOPs $\downarrow$
FIQA
SupportNet-M4 [ 15 ]	52.70	0.6545	81.46	2.15M	73.2K
KeyNet-M4 [ 15 ]	51.52	0.6434	80.26	2.30M	29.8K
GradNet 144-M8	63.71	0.7553	90.83	442K	28.5K
GradNet 128-M9	63.87	0.7562	91.14	442K	29.3K
GradNet 112-M10	64.25	0.7591	91.06	430K	29.3K

Table 2: Retrieval performance and efficiency under the restricted (no-noise) setting (25K training steps).

Model	Acc. $\uparrow$	MRR $\uparrow$	[email protected]% $\uparrow$	Params $\downarrow$	Proj. FLOPs $\downarrow$
FIQA
SupportNet-M4 [ 15 ]	51.76	0.6449	80.53	2.15M	73.2K
KeyNet-M4 [ 15 ]	63.20	0.7354	86.60	2.30M	29.8K
GradNet 144-M8	64.50	0.7618	91.52	442K	28.5K
GradNet 128-M9	64.19	0.7592	91.43	442K	29.3K
GradNet 112-M10	64.60	0.7614	91.28	430K	29.3K

Table 3: Retrieval performance and efficiency with noise-augmented training using 10 noisy samples per query (10K training steps).

Function Approximation

See all 34 papers

Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients
May 3, 2026Sejun Park, Yeachan Park, Geonho HwangAutomatic Differentiation Neural Network Approximation Theory
Department of Artificial Intelligence, Korea University, Seoul, 02841, Republic of Korea · Department of Mathematics and Statistics, Sejong University, Seoul, 05006, Republic of Korea · Department of Mathematical Sciences, Gwangju Institute of Science and Technology, Gwangju, 61005, Republic of Korea
Worker Disagreement Reveals Sharp Directions in Local SGD
May 26, 2026Tolga Dimlioglu, Kristi Topollai, Anna ChoromanskaNeural Network Training Dynamics Loss Landscape Geometry
New York University, New York, USA
Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks
Feb 26, 2026Wenquan Ma, Yang Sui, Jiaye Teng +3Generalization Bounds Algorithmic Generalization

May 3, 2026J/K move · Enter open · S save

Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

Sejun Park, Yeachan Park, Geonho Hwang

Department of Artificial Intelligence, Korea University, Seoul, 02841, Republic of Korea · Department of Mathematics and Statistics, Sejong University, Seoul, 05006, Republic of Korea · Department of Mathematical Sciences, Gwangju Institute of Science and Technology, Gwangju, 61005, Republic of Korea

Automatic Differentiation Neural Network Approximation Theory

Open paper AskPDF arXiv

Theoretical studies show that for any differentiable function on a compact domain, there exists a neural network that approximates both the function values and gradients. However, such a result cannot be used in practice since it assumes real parameters and exact internal operations. In contrast, real implementations only use a finite subset of reals and machine operations with round-off errors. In this work, we investigate whether a similar result holds for neural networks under floating-point arithmetic, when the gradient with respect to the input is computed by the automatic differentiation algorithm

D^\mathtt{AD}

. We first show that given a floating-point function

φ

(e.g., a loss function), arbitrary function values and gradients can be represented by a floating-point network

f

and

D^\mathtt{AD}(φ\circ f)

, respectively. We further extend this result: given

φ_1,\dots,φ_n

D^\mathtt{AD}(φ_i\circ f)

can simultaneously represent arbitrary gradients while

f

represents the target values, under mild conditions. Our results hold for practical activation functions, e.g.,

\mathrm{ReLU}

\mathrm{ELU}

\mathrm{GeLU}

\mathrm{Swish}

\mathrm{Sigmoid}

, and

\mathrm{tanh}

Abstract

Function Approximation

Explore similar work

Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

Worker Disagreement Reveals Sharp Directions in Local SGD

Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks