Bilevel Optimization of Topology and Hyperparameters (BOTH)

Suryanarayanan Manoj Sanu; Miguel Anibal Bessa; Alejandro Marcos Aragón

cs.LGSep 18, 2026

Bilevel Optimization of Topology and Hyperparameters (BOTH)

Authors: Suryanarayanan Manoj Sanu, Miguel Anibal Bessa, Alejandro Marcos Aragón

Organizations: Faculty of Mechanical Engineering Delft university of technology Delft The Netherlands · School of engineering Brown University Providence United States of America

Abstract

Topology optimization (TO) represents a significant step towards automating the design process: given a working simulation, TO can produce a viable prototype at the press of a button by differentiating the simulation and iteratively improving the design. In practice, however, TO is riddled with magic numbers''---hyperparameters whose tuning significantly affects the outcome. Finding the right values typically requires not only deep problem-specific knowledge but also extensive trial-and-error. While practitioners can use surrogate-assisted hyperparameter optimization as an alternative, this approach requires strictly limiting the number of hyperparameters through careful problem formulation. Here, we propose differentiating TO itself using automatic differentiation. This yields hypergradients'' that allow us to tune these hyperparameters in tandem with the primary optimization. We show that evaluating just one or two steps of TO is sufficiently informative and that the method scales favorably to thousands of hyperparameters at an expense comparable to only a few standard TO runs. We demonstrate this approach on stress-constrained and compliance problems, with the latter utilizing a neural parameterization of the density field.

Figures & tables

Figure 1: Reverse-mode automatic differentiation of the hypergradient. Top: forward computational graph for the bilevel problem, showing each optimizer step

\boldsymbol{{\psi}}_{j}

as a node taking both the previous iterate

\boldsymbol{x}_{j-1}

and the UL variables

\boldsymbol{y}

as inputs, with the UL objective

F

evaluated at the final iterate

\boldsymbol{x}_{t}

. Bottom: reverse-mode backward pass. A unit seed enters at

F

and propagates backward through the trajectory (blue arrows) via the recursion

\boldsymbol{e}_{j-1}=\boldsymbol{e}_{j}\mathbf{A}_{j}

. At each step, a second branch (red arrows) contributes

\boldsymbol{e}_{j}\mathbf{B}_{j}

to the register, where the hypergradient is accumulated as

\boldsymbol{c}_{j-1}=\boldsymbol{c}_{j}+\boldsymbol{e}_{j}\mathbf{B}_{j}

. The final hypergradient

dF/d\boldsymbol{y}

is read off as

\boldsymbol{c}_{0}

at the leftmost endpoint of the register.

Figure 2: Illustration of challenges in gradient-based bilevel optimization using the SMD-1 benchmark with two upper-level (UL) and two lower-level (LL) variables. (a) True UL objective landscape (solid contours), obtained using the analytical LL optimum at each UL point, overlaid with an approximate landscape (dashed) obtained from a single Adam step at the LL from a random initialization. The true optimum at

(0,0)

and the shifted apparent optimum are marked. (b) Short-horizon bias: shift in the UL minimum from the true location

y_{2}^{\star}

to the apparent minimum

y_{2}^{t}

when the LL is run for

t

steps, shown for several LL initializations. (c) Cosine similarity between the estimated hypergradients (pseudo first-order approximation and vanilla unrolling) and the true hypergradient; dashed lines show individual runs and solid lines show medians. (d) Ratio of the

L^{2}

norms of the estimated and true hypergradients, illustrating gradient norm explosion at large LL horizons.

Figure 3: Effect of warmstarting on the bilevel optimization of SMD-1. All four panels compare warmstarting and coldstarting under otherwise identical settings (Adam at both levels, gradient clipping, annealing schedule). Results are aggregated over

72

runs combining

9

UL and

8

LL initializations; solid lines show medians and shaded bands show one standard deviation. (a) Convergence history of the UL objective; the horizontal axis is the cumulative number of LL steps. (b) Fraction of UL iterations spent at each value of the LL horizon

t

, illustrating how the annealing schedule allocates the budget. For each scheme, the bar heights sum to one. (c) Euclidean distance between the UL iterate

\boldsymbol{y}^{(k)}

after

k

UL iterations and the true UL optimum

\boldsymbol{y}^{\star}=(0,0)

. (d) LL optimality measured by the norm of the LL objective gradient

\|\nabla_{\boldsymbol{x}}f\|

at the final LL iterate.

Figure 4: The UL landscape for neural topology optimization with a CNN architecture. (a) Landscape obtained by independently solving the lower-level problem from the same fixed initialization for each hyperparameter pair

(\eta,\beta)

, corresponding to the landscape observed by BO. (b) Effective landscape encountered by BOTH under warmstarted optimization, illustrating the path-dependent nature of hypergradient-based tuning; the red asterisk marks the current UL iterate and the trailing line shows the trajectory followed.. Here,

k

denotes the UL iteration index and

t

denotes the number of LL optimization steps performed at that iteration.

Figure 5: Comparison between Bayesian optimization (BO) and our method (BOTH) for neural topology optimization with a CNN architecture. (a) Convergence histories as a function of cumulative number of linear solves used by each method. Dashed lines show individual runs from different initializations; the bold line shows the median. (b) Decomposition of the final UL objective into its compliance and discreteness components across runs, with medians marked by horizontal lines. (c) Typical topologies obtained for the MBB beam together with their UL objective values, where $c^{\star}$ denotes compliance (evaluated at $p=1$ ) and $\mathcal{D}$ measures discreteness. (d) Evolution of the four tuned hyperparameters: learning rate $\eta$ , Heaviside sharpness $\beta$ , SIMP penalty exponent $p$ , and filter radius $r_{\min}$ ; $\eta$ is shown on the right $y$ -axis. (e) Mean number of UL iterations (normalized against the total UL iterations) spent at each LL horizon $t$ , averaged across all runs, with one standard deviation shown as error bars; $k$ denotes the UL iteration index. (f) LL objective value at the final LL iterate $f(\boldsymbol{x}_{t}^{(k)})$ as a function of UL iteration $k$ .

Figure 6: Scaling to a high-dimensional UL via per-element projection sharpness. Each element is assigned its own $\beta_{e}$ , giving 6915 UL hyperparameters (high-D, green), compared against the earlier single- $\beta$ case (low-D, gray), each shown for two UL Adam learning rates. The high-D and low-D cases converge comparably at similar per-iteration cost. Bottom: a representative final design and the corresponding learned $\beta_{e}$ field.

Figure 7: Two-bar stress-constrained optimization benchmark problem, borrowed from [ 10 ] . (top) Schematic of the structure and boundary conditions for the two dimensional toy problem, with the areas of the bars

{\bf\it a}=(a_{1},a_{2})

being the design variables. Load is applied axially at the point of contact. (a) The problem’s design space, outlining the two vanishing stress constraints (

\bar{g}_{1},\bar{g}_{2}

). Note that the constraint boundaries also include the coordinate axes. The main feasible region (in white) has an one-dimensional extension along the

a_{1}

axis (hashed) allowing access to the singular global optimum at

(1,0)

. (b) The augmented objective of the quadratic penalty method, consisting of the mass and the sum of the squares of the constraints for different values of the penalty factor (

\mu

) is shown. (c) The equivalent relaxed constraints (using the optimal Lagrange multipliers at

(1,0)

) for two values of

\mu

Figure 8: Sensitivity of the quadratic penalty method (QPM) to hyperparameters on the benchmark two-bar stress-constrained optimization problem. The original constrained problem is converted via QPM into an unconstrained formulation and solved by Adam, with the penalty

\mu

incremented following

\mu^{(k)}=\mu_{\text{inc}}\mu^{(k-1)}

. Four trajectories are shown, corresponding to two values of

\mu_{\text{inc}}

paired with two learning rates

\eta=\left\{0.01,0.02\right\}

. (a) Convergence of the objective: only one trajectory reaches the globally optimum

f\left({\bf\it a}_{\star}\right)=0.6

; (b) Evolution of the constraints for each of the runs, with feasible values

\bar{g}_{i}=0

(One constraint with dashes and other with dots); (c) Trajectories in the design space

\boldsymbol{a}

, with constraint boundaries shown in black lines.

Figure 9: Results for the two-dimensional stress problem with standard parameterization. (a) Each dot represents one optimization run from a distinct LL initialization; green indicates convergence to the global optimum, gray indicates convergence to the local minimum. (b, c) Convergence histories of the objective and the two constraints for three representative runs (A, B, C). (d, e) Learned schedules for the LL learning rate

\eta

and the QPM penalty

\mu

(both parameterized in log scale) along these three runs. (f) Trajectories projected into the

(a_{1},a_{2})

space.

Figure 10: Boundary conditions for the L-shaped bracket problem. The top edge is fully fixed, and a downward force is distributed over a non-designable solid region (orange square). A padding layer (hatched) of non-designable void elements surrounds the domain and is included in the FEA, so that internal and external boundaries are treated consistently [ 67 ] . The designable elements lie within the L-shaped domain.

Figure 11: Results for tuning hyperparameters in the design of an L-shaped beam under stress constraints. (a) Convergence of the two components of the upper-level loss, the objective (mass) and the penalty term (sum of squared constraints), against the number of linear solves consumed by the bilevel optimization. (b) Final design, projected with $\beta=64$ . (c) Ratio of von Mises stress $\sigma$ to the stress limit $\sigma_{\max}$ on elements with density above $0.5$ , evaluated assuming solid material. (d) Evolution of the lower-level learning rate and of the penalty factor $\mu$ of the quadratic penalty method. (e) Evolution of the projection parameter $\beta$ and the radius of the density filter. (f) Distribution of the upper-level constraint values at the start and end of the bilevel optimization.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Non-convexity exposure of the LL trajectory in the modified bilevel problem of Eqn. ( 38 ). Several bilevel optimizations are run from randomly sampled UL–LL initializations with a fixed LL horizon

t=40

, differing only in the LL initialization scheme (coldstart or warmstart). At each UL iteration

k

, pairs of points are sampled from the LL trajectory and the non-convexity ratio (Eqn. ( 40 )) is computed; results are aggregated across all runs. (a) An example LL landscape showing multiple local minima. (b) Non-convexity ratio versus UL iteration

k

for coldstart and warmstart; shaded regions show one standard deviation across runs. (c) Mean pairwise distance within the LL trajectory at each UL iteration.

UL variable	$y_{\min}$	$y_{\max}$
Learning rate $\eta$	$10^{-8}$	$10^{-1.5}$
Heaviside projection $\beta$	$0.05$	$15.0$
Filter radius $r_{\min}$	$1.5$	$10.0$
SIMP penalty $p$	$1.0$	$10.0$

Appendix

Table 1: Bounds on UL variables for hyperparameter tuning experiment.

Figure 13: The UL landscape for standard topology optimization. (a) Landscape obtained by independently solving the lower-level problem from the same fixed initialization for each hyperparameter pair

(\eta,\beta)

, corresponding to the landscape observed by BO. (b) Effective landscape encountered by BOTH under warmstarted optimization; the red asterisk marks the current UL iterate and the trailing line shows the trajectory followed. Here,

k

denotes the UL iteration index and

t

denotes the number of LL optimization steps performed at that iteration.

Figure 14: Ablation study for compliance minimization with neural reparameterization, varying one component of the base configuration at a time across

6

BOTH runs; run-to-run variation arises solely from differences in network or hyperparameter initialization. (a) Effect of the weighting coefficient

k

in the UL objective, which controls the relative contribution of compliance and discreteness. (b) Effect of the Heaviside sharpness

\beta

used in the UL objective;

\beta=\beta_{\text{LL}}

indicates that the LL value of

\beta

is reused at the UL, introducing a direct hypergradient term. (c) Effect of the initialization scheme: warmstarting versus coldstarting. (d) Effect of the initial LL learning rate

\eta^{(0)}

, normally selected by the probe procedure but set manually here.

Hypergradient Optimization

See all 11 papers

Sep 23, 2026

KATOsuper: Surrogate-accelerated neural topology optimization with sensitivity-consistent Fourier neural operators

Topology optimization (TO) remains computationally intensive due to repeated finite element analysis (FEA) evaluations required at each iteration. While neural network-based surrogates offer potential acceleration, existing approaches often suffer from gradient inconsistency between predicted objectives and sensitivities, leading to optimization instability. This work presents KATOsuper, an objective-agnostic framework that couples neural-reparameterized topology optimization with a Sensitivity-Consistent Fourier Neural Operator (SC-FNO). The framework employs the forward_split architecture, which derives deployed sensitivities via automatic differentiation through the predicted objective field and thereby preserves consistency between the predicted objective and the gradient used for optimization. The case studies include three 2D benchmark problems and three 3D structures considering compliance or stress minimization. A physics-informed multi-channel input encoding with Fourier position embedding enables resolution-invariant learning, supporting zero-shot extrapolation beyond the training resolution, with useful performance at moderate scaling factors and topology-preserving exploration at up to 64x without retraining. The framework extends to 3D through KATO3D, featuring novel KANConv3D blocks with learnable B-spline activations. KATOsuper demonstrates 15--110x deployment-time speedup over MATLAB baselines while maintaining competitive optimality, with the clearest gains observed in complex 3D and stress-optimization cases. The insight that sensitivity direction matters more than magnitude enables robust optimization even with approximate physics evaluation, extensible to other differentiable physics-driven design objectives.

Shengyu Yan, Jasmin Jelovica

Department of Mechanical Engineering, The University of British Columbia, 6250 Applied Science Ln, Vancouver, V6T 1Z4, Canada · Department of Civil Engineering, The University of British Columbia, 6250 Applied Science Ln, Vancouver, V6T 1Z4, Canada

Neural Surrogate Modeling Topology Optimization Surrogate Modeling Fourier Neural Operator Surrogate-Assisted Optimization

Ask PDF

Jun 1, 2026

On the Generalization in Topology Optimization via Sensitivity-Conditioned Bernoulli Flow Matching

Surrogate models for topology optimization (TO) exhibit highly variable out-of-distribution (OOD) generalization under distribution shifts such as changing loads or boundary conditions, yet the source of this variability remains unclear. We hypothesize that OOD performance is governed by how much information the conditioning signal preserves about the adjoint sensitivity (reduced gradient) that drives classical TO. Modeling the TO pipeline as a causal Markov chain, the Data Processing Inequality establishes that, under this abstraction, the sensitivity field is an information-theoretically optimal conditioning signal for topology prediction. However, computing exact adjoint sensitivities can be expensive or unavailable in practice; we observe that certain physical fields can approximate sensitivities through monotone transformations. To formalize this, we introduce pseudo-sensitivities to characterize which fields enable generalization versus those that are information-poor. We then show that a sensitivity-conditioned Bernoulli flow-matching generator empirically confirms these predictions: conditioning on sensitivities yields state-of-the-art OOD performance, while increasingly distant physical fields degrade toward raw parameter conditioning. Results hold across structural TO benchmarks under load shifts and our new CFD-TO dataset under boundary-condition shifts such as multi-outlet configurations. Code and datasets are available at https://tum-pbs.github.io/topotransformer/ .

Mohammad Rashed, Duarte F. Valoroso Madeira, Babak Gholami +3

School of Computation, Information and Technology, Technical University of Munich, Germany · BMW AG, Munich, Germany · Munich Center for Machine Learning, Munich, Germany. +1

Neural Surrogate Modeling Topology Optimization OOD Generalization Sensitivity Analysis

Ask PDF

Jun 18, 2026

eCNNTO: A Highly Generalizable ConvNet for Accelerating Topology Optimization

This work proposes an element-based Convolutional Neural Network (CNN) to accelerate density-based Topology Optimization (TO), termed eCNNTO. TO generally undergoes a large number of iterations, where finite element analysis is performed in every iteration, leading to the efficiency bottleneck especially when dense meshes are used to achieve high-resolution designs. To address this limitation, eCNNTO is proposed to build upon Kallioras et al. (2020), where a Deep Belief Network (DBN) was trained for every element to predict its near-optimal density from its early history, thereby skipping the great majority of iterations and significantly accelerating the TO procedure. However, the method lacks spatial correlations among neighboring elements and may lead to disconnected features in the final structure. The proposed method employs CNN with residual connections to address this issue. On top of it, a novel training strategy is introduced to further enhance the optimization efficiency, where the training dataset consists of the final stage density histories rather than early ones. This change can also help reduce the required training data size. eCNNTO requires only a small dataset to train and yet it can be generalized to problems with largely different boundary conditions, loading cases, design domain geometries, mesh resolutions, as well as non-design domains. In the end, the generalization capabilities and efficiency of eCNNTO are demonstrated through a variety of examples in two and three dimensions, achieving up to 90% and 97% reduction of iterations, respectively.

Shengbiao Lu, Xiaodong Wei

Global college, Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai, 200240, Shanghai, China

Topology Optimization Deep Learning Optimization Convolutional Neural Networks

Ask PDF

Abstract

Hypergradient Optimization

Explore similar work

KATOsuper: Surrogate-accelerated neural topology optimization with sensitivity-consistent Fourier neural operators

On the Generalization in Topology Optimization via Sensitivity-Conditioned Bernoulli Flow Matching

eCNNTO: A Highly Generalizable ConvNet for Accelerating Topology Optimization