We study the numerical resolution of uniformly elliptic Dirichlet boundary value problems (BVPs) by minimization of the PDE residual over a hypothesis space of Neural Networks made to satisfy exactly the boundary conditions through multiplication by a user-defined
distance-to-boundary approximation ρ. While an
L2(∂Ω) boundary penalty cannot generally guarantee
H2(Ω) error control, we show that exact enforcement can, provided
ρ is chosen appropriately. More precisely, we identify a sharp sufficient condition on
ρ preserving the regularity needed for
H2(Ω) error control, provide practical constructions of such
ρ for general smooth domains
Ω, and refer to the resulting networks as
boundary-adapted. For boundary-adapted Rectified Quadratic Unit (ReQU) and hyperbolic tangent (tanh) networks, we then derive explicit high-probability
H2(Ω) convergence rates by combining approximation-theoretic and statistical-learning arguments, making explicit the dependence on
ρ. Numerical experiments support our theoretical results, showing that appropriate choices of
ρ improve accuracy and convergence, while poorly chosen distance functions can substantially degrade the solution. Our proof also yields new VC-dimension bounds for hypothesis spaces of higher-order derivatives of ReQU and tanh networks, together with new approximation bounds for shallow ReQU networks in higher-order Sobolev norms, all of which are of important independent interest.