cs.LGSep 18, 2026
SaveBilevel Optimization of Topology and Hyperparameters (BOTH)
Organizations: Faculty of Mechanical Engineering Delft university of technology Delft The Netherlands · School of engineering Brown University Providence United States of America
Abstract
Topology optimization (TO) represents a significant step towards automating the design process: given a working simulation, TO can produce a viable prototype at the press of a button by differentiating the simulation and iteratively improving the design. In practice, however, TO is riddled with magic numbers''---hyperparameters whose tuning significantly affects the outcome. Finding the right values typically requires not only deep problem-specific knowledge but also extensive trial-and-error. While practitioners can use surrogate-assisted hyperparameter optimization as an alternative, this approach requires strictly limiting the number of hyperparameters through careful problem formulation. Here, we propose differentiating TO itself using automatic differentiation. This yields hypergradients'' that allow us to tune these hyperparameters in tandem with the primary optimization. We show that evaluating just one or two steps of TO is sufficiently informative and that the method scales favorably to thousands of hyperparameters at an expense comparable to only a few standard TO runs. We demonstrate this approach on stress-constrained and compliance problems, with the latter utilizing a neural parameterization of the density field.
Figures & tables
Figure 1: Reverse-mode automatic differentiation of the hypergradient. Top: forward computational graph for the bilevel problem, showing each optimizer step as a node taking both the previous iterate and the UL variables as inputs, with the UL objective evaluated at the final iterate . Bottom: reverse-mode backward pass. A unit seed enters at and propagates backward through the trajectory (blue arrows) via the recursion . At each step, a second branch (red arrows) contributes to the register, where the hypergradient is accumulated as . The final hypergradient is read off as at the leftmost endpoint of the register.
Figure 2: Illustration of challenges in gradient-based bilevel optimization using the SMD-1 benchmark with two upper-level (UL) and two lower-level (LL) variables. (a) True UL objective landscape (solid contours), obtained using the analytical LL optimum at each UL point, overlaid with an approximate landscape (dashed) obtained from a single Adam step at the LL from a random initialization. The true optimum at and the shifted apparent optimum are marked. (b) Short-horizon bias: shift in the UL minimum from the true location to the apparent minimum when the LL is run for steps, shown for several LL initializations. (c) Cosine similarity between the estimated hypergradients (pseudo first-order approximation and vanilla unrolling) and the true hypergradient; dashed lines show individual runs and solid lines show medians. (d) Ratio of the norms of the estimated and true hypergradients, illustrating gradient norm explosion at large LL horizons.
Figure 3: Effect of warmstarting on the bilevel optimization of SMD-1. All four panels compare warmstarting and coldstarting under otherwise identical settings (Adam at both levels, gradient clipping, annealing schedule). Results are aggregated over runs combining UL and LL initializations; solid lines show medians and shaded bands show one standard deviation. (a) Convergence history of the UL objective; the horizontal axis is the cumulative number of LL steps. (b) Fraction of UL iterations spent at each value of the LL horizon , illustrating how the annealing schedule allocates the budget. For each scheme, the bar heights sum to one. (c) Euclidean distance between the UL iterate after UL iterations and the true UL optimum . (d) LL optimality measured by the norm of the LL objective gradient at the final LL iterate.
Figure 4: The UL landscape for neural topology optimization with a CNN architecture. (a) Landscape obtained by independently solving the lower-level problem from the same fixed initialization for each hyperparameter pair , corresponding to the landscape observed by BO. (b) Effective landscape encountered by BOTH under warmstarted optimization, illustrating the path-dependent nature of hypergradient-based tuning; the red asterisk marks the current UL iterate and the trailing line shows the trajectory followed.. Here, denotes the UL iteration index and denotes the number of LL optimization steps performed at that iteration.
Figure 5: Comparison between Bayesian optimization (BO) and our method (BOTH) for neural topology optimization with a CNN architecture. (a) Convergence histories as a function of cumulative number of linear solves used by each method. Dashed lines show individual runs from different initializations; the bold line shows the median. (b) Decomposition of the final UL objective into its compliance and discreteness components across runs, with medians marked by horizontal lines. (c) Typical topologies obtained for the MBB beam together with their UL objective values, where denotes compliance (evaluated at ) and measures discreteness. (d) Evolution of the four tuned hyperparameters: learning rate , Heaviside sharpness , SIMP penalty exponent , and filter radius ; is shown on the right -axis. (e) Mean number of UL iterations (normalized against the total UL iterations) spent at each LL horizon , averaged across all runs, with one standard deviation shown as error bars; denotes the UL iteration index. (f) LL objective value at the final LL iterate as a function of UL iteration .
Figure 6: Scaling to a high-dimensional UL via per-element projection sharpness. Each element is assigned its own , giving 6915 UL hyperparameters (high-D, green), compared against the earlier single- case (low-D, gray), each shown for two UL Adam learning rates. The high-D and low-D cases converge comparably at similar per-iteration cost. Bottom: a representative final design and the corresponding learned field.
Figure 7: Two-bar stress-constrained optimization benchmark problem, borrowed from [ 10 ] . (top) Schematic of the structure and boundary conditions for the two dimensional toy problem, with the areas of the bars being the design variables. Load is applied axially at the point of contact. (a) The problem’s design space, outlining the two vanishing stress constraints ( ). Note that the constraint boundaries also include the coordinate axes. The main feasible region (in white) has an one-dimensional extension along the axis (hashed) allowing access to the singular global optimum at . (b) The augmented objective of the quadratic penalty method, consisting of the mass and the sum of the squares of the constraints for different values of the penalty factor ( ) is shown. (c) The equivalent relaxed constraints (using the optimal Lagrange multipliers at ) for two values of .
Figure 8: Sensitivity of the quadratic penalty method (QPM) to hyperparameters on the benchmark two-bar stress-constrained optimization problem. The original constrained problem is converted via QPM into an unconstrained formulation and solved by Adam, with the penalty incremented following . Four trajectories are shown, corresponding to two values of paired with two learning rates . (a) Convergence of the objective: only one trajectory reaches the globally optimum ; (b) Evolution of the constraints for each of the runs, with feasible values (One constraint with dashes and other with dots); (c) Trajectories in the design space , with constraint boundaries shown in black lines.
Figure 9: Results for the two-dimensional stress problem with standard parameterization. (a) Each dot represents one optimization run from a distinct LL initialization; green indicates convergence to the global optimum, gray indicates convergence to the local minimum. (b, c) Convergence histories of the objective and the two constraints for three representative runs (A, B, C). (d, e) Learned schedules for the LL learning rate and the QPM penalty (both parameterized in log scale) along these three runs. (f) Trajectories projected into the space.
Figure 10: Boundary conditions for the L-shaped bracket problem. The top edge is fully fixed, and a downward force is distributed over a non-designable solid region (orange square). A padding layer (hatched) of non-designable void elements surrounds the domain and is included in the FEA, so that internal and external boundaries are treated consistently [ 67 ] . The designable elements lie within the L-shaped domain.
Figure 11: Results for tuning hyperparameters in the design of an L-shaped beam under stress constraints. (a) Convergence of the two components of the upper-level loss, the objective (mass) and the penalty term (sum of squared constraints), against the number of linear solves consumed by the bilevel optimization. (b) Final design, projected with . (c) Ratio of von Mises stress to the stress limit on elements with density above , evaluated assuming solid material. (d) Evolution of the lower-level learning rate and of the penalty factor of the quadratic penalty method. (e) Evolution of the projection parameter and the radius of the density filter. (f) Distribution of the upper-level constraint values at the start and end of the bilevel optimization.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Non-convexity exposure of the LL trajectory in the modified bilevel problem of Eqn. ( 38 ). Several bilevel optimizations are run from randomly sampled UL–LL initializations with a fixed LL horizon , differing only in the LL initialization scheme (coldstart or warmstart). At each UL iteration , pairs of points are sampled from the LL trajectory and the non-convexity ratio (Eqn. ( 40 )) is computed; results are aggregated across all runs. (a) An example LL landscape showing multiple local minima. (b) Non-convexity ratio versus UL iteration for coldstart and warmstart; shaded regions show one standard deviation across runs. (c) Mean pairwise distance within the LL trajectory at each UL iteration.
| UL variable | ||
|---|---|---|
| Learning rate | ||
| Heaviside projection | ||
| Filter radius | ||
| SIMP penalty |
Table 1: Bounds on UL variables for hyperparameter tuning experiment.
Figure 13: The UL landscape for standard topology optimization. (a) Landscape obtained by independently solving the lower-level problem from the same fixed initialization for each hyperparameter pair , corresponding to the landscape observed by BO. (b) Effective landscape encountered by BOTH under warmstarted optimization; the red asterisk marks the current UL iterate and the trailing line shows the trajectory followed. Here, denotes the UL iteration index and denotes the number of LL optimization steps performed at that iteration.
Figure 14: Ablation study for compliance minimization with neural reparameterization, varying one component of the base configuration at a time across BOTH runs; run-to-run variation arises solely from differences in network or hyperparameter initialization. (a) Effect of the weighting coefficient in the UL objective, which controls the relative contribution of compliance and discreteness. (b) Effect of the Heaviside sharpness used in the UL objective; indicates that the LL value of is reused at the UL, introducing a direct hypergradient term. (c) Effect of the initialization scheme: warmstarting versus coldstarting. (d) Effect of the initial LL learning rate , normally selected by the probe procedure but set manually here.