We study the width required for a randomly initialized hidden layer of a neural network to achieve rank lifting. Namely, given a dataset
X∈Rm×d of
m,
d-dimensional input vectors separated by an angle of at least
θ, we consider the random feature matrix
σ(XR), where
R is standard Gaussian. For positively homogeneous nonpolynomial activations, which include sign, Heaviside, ReLU, and ReLU powers among others, we prove that
n≳θ1max{m,log(δ1)} neurons suffice for
σ(XR) to have full row rank
m with probability at least
1−δ. This dimension-free bound exponentially improves the previous general-dimensional guarantee for sign features (Drago et al., 2026) and is essentially tight. The proof shows that one random feature column escapes every proper subspace of
Rm with probability
Ω(θ), using a coupling of nearby Gaussian directions and a local crossing of the induced hyperplane arrangement. We also study stable rank lifting, where the goal is to establish a quantitative analogue of exact rank lifting, i.e., a lower bound on the smallest eigenvalue of the empirical feature Gram matrix in high-probability. Our analysis unifies and generalizes stable rank guarantees for all
q-homogeneous non-polynomial activations following prior work in Panigrahi et al. (2020) and Song (2026). In particular, we combine a diagonally dominant Taylor tail of the population kernel with truncation and matrix concentration, to show that for positively homogeneous nonpolynomial activations, stable rank lifting is achieved at width
n≳Cqθ2q+1mlog2q+21(θm)log(δm), where
q is the degree of the activation and
C>0 is some universal constant.