Optimization, Optimal Control, and Reinforcement Learning for Robotics

Yonghyeon Lee

cs.ROOct 23, 2026

Optimization, Optimal Control, and Reinforcement Learning for Robotics

Authors: Yonghyeon Lee

Organizations: Department of Artificial Intelligence, Yonsei University

Abstract

These lecture notes provide an introduction to optimization, optimal control, and reinforcement learning for robotics, emphasizing their mathematical connections and practical relevance. They present established material compiled and edited by Yonghyeon Lee, rather than new research results. The notes were developed through a human and AI collaborative writing process, with AI assistance in drafting, organizing, and refining the exposition. This first version is a preliminary collection of notes, not yet a fully developed textbook, and currently contains no figures or graphs. The notes may be revised, corrected, and expanded in future versions.

Figures & tables

Aspect	Indirect method	Direct method
Formulated from	PMP necessary conditions	original dynamics, cost, and constraints
Typical unknowns	costate boundary values, multipliers, switching times	sampled controls and, in transcription, sampled states
Numerical task	root finding or a boundary-value solve	nonlinear minimization
Path constraints	require extra multipliers, arcs, and switching conditions	imposed directly as finite-dimensional constraints

Table 9.1: Indirect and direct numerical optimal control.

Method	Main variables	Dynamics during iterations	Typical feature
Single shooting	controls	exactly satisfied	compact but sensitive
Multiple shooting	controls and segment states	segmentwise satisfied	balanced conditioning
Direct transcription	states and controls	enforced as constraints	sparse and constraint-friendly
iLQR/DDP	nominal controls and local policy	satisfied by rollout	fast structured local solve

Table 9.2: Common trajectory-optimization formulations.

Panel of Figure 10 — Figure 10.1: The MPC feedback loop. Only the first optimized control is executed before the state estimate and plan are updated.

Device	Useful work	Main qualification
CPU	small models, branching logic, parallel simulator instances	a small batch may not justify accelerator overhead
GPU	batched rollouts, compatible physics, neural inference, reductions	memory traffic and sequential horizon steps still matter
NPU	supported neural dynamics or perception inference	neural inference support does not imply support for the entire planner

Table 11.1: Typical opportunities for parallel sampling-based MPC. The appropriate device depends on the workload, not only its peak throughput.

Optimal Control

Optimal control studies how to choose actions over time to optimize performance while satisfying system dynamics and constraints.

See all 139 papers

An Introduction to Zero-Order Optimization Techniques for Robotics
Jun 27, 2025Armand Jordana, Jianghan Zhang, Joseph Amigo +1Zeroth-Order Optimization Robotic Optimization
Machines in Motion Laboratory, New York University, USA
Foundations of Reinforcement Learning and Control:Connections and New Perspectives
Aug 3, 2026Claire Vernade, Onno Eberhard, Martha White +4RL Control Data-Driven Control
University of Technology Nuremberg · Max Planck Institute for Intelligent Systems, Tübingen, Germany · University of Alberta and Alberta Machine Intelligence Institute (Amii) +2
Ergodic Control and Controlled Diffusion for Robot Learning: Review and Tutorial
Sep 9, 2026Max Muchen Sun, Cem Bilaloglu, Ananya Rao +6Ergodic Control Robot Learning
Northwestern University · Idiap Research Institute · Carnegie Mellon University +4

Jun 27, 2025J/K move · Enter open · S save

An Introduction to Zero-Order Optimization Techniques for Robotics

Armand Jordana, Jianghan Zhang, Joseph Amigo, Ludovic Righetti

Machines in Motion Laboratory, New York University, USA

Zeroth-Order Optimization Robotic Optimization Robotic RL Trajectory Optimization

Open paper AskPDF arXiv

Zero-order optimization techniques are becoming increasingly popular in robotics due to their ability to handle non-differentiable functions and escape local minima. These advantages make them particularly useful for trajectory optimization and policy optimization. In this work, we propose a mathematical tutorial on random search. It offers a simple and unifying perspective for understanding a wide range of algorithms commonly used in robotics. Leveraging this viewpoint, we classify many trajectory optimization methods under a common framework and derive novel competitive RL algorithms.

Abstract

Optimal Control

Explore similar work

An Introduction to Zero-Order Optimization Techniques for Robotics

Foundations of Reinforcement Learning and Control:Connections and New Perspectives

Ergodic Control and Controlled Diffusion for Robot Learning: Review and Tutorial