ECE 517: Lecture Schedule

Tue Aug 23
[🗒 scribbles]
[🎥 video]

Introduction and administrivia
Adaptation as control in the presence of large-scale uncertainty
Some history: gain scheduling, model reference adaptive control, Lyapunov redesign, self-tuning controllers, direct vs. indirect adaptive control
Basic example: adaptive output regulation of a first-order scalar linear system
Lyapunov stability (review)
Certainty equivalence principle

Thu Aug 25
[🗒 scribbles]
[🎥 video]

Stability
Weak Lyapunov functions
Krasovskii-LaSalle theorem
Barbalat's lemma
Adaptive regulation revisited

Tue Aug 30

Review: weak Lyapunov functions
Connections to observability: Uniform Complete Observability (Kalman)
Adaptive regulation revisited: alternative analysis with V(x) = x²/2 using boundedness and Barbalat's lemma
Universal regulators: definition
Example: scalar linear plant dy/dt = a y + b u, known vs. unknown sign of b

Thu Sep 1

Universal regulation for linear time-invariant systems
Scalar systems:

• fundamental limits (controllers with rational dynamics cannot achieve universal regulation)
• achievability of universal regulation via Nussbaum gains, exploration vs. exploitation

Tue Sep 6

Universal regulation for time-invariant linear systems, continued
Higher-dimensional systems:

• preliminaries (relative degree, minimum phase) -- see Appendix A in Sepulchre et al., Constructive Nonlinear Control
• achievability of universal regulation for SISO systems via output feedback with Nussbaum gains

Thu Sep 8

Introduction to Lyapunov-based design
Universal regulation example revisited
Control Lyapunov functions and continuous selection problem
Sontag's universal formula for control-affine systems
Comparison and contrast with other stabilizing controllers (e.g., feedback linearization)
Application to adaptive stabilization of a first-order scalar linear plant

Tue Sep 13

Backstepping
Motivation and procedure for control-affine systems
Comparison with feedback linearization

Thu Sep 15

Adaptive backstepping
Effect of unknown parameters and iterative tuning function update
Introduction to parameter estimation

Tue Sep 20

Optimization in continuous time
Global minima, first-order optimality condition
Gradient flow
Convex and strongly convex functions
Lyapunov analysis

Thu Sep 22

Online optimization in continuous time, Part 1
Motivation through parameter estimation in adaptive control
The online optimization paradigm: asymptotic no-regret property
No-regret in online convex optimization via gradient flow
Analysis for uniformly Lipschitz costs via Lyapunov functions
Convergence criteria for costs and parameter estimates via Barbalat
Analysis for uniformly strongly convex costs
Convergence criteria for costs and parameter estimates: persistency of excitation as integrated strong convexity

Tue Sep 27
[🗒 scribbles]
[🎥 video]

Online optimization in continuous time, Part 2
Extension to unbounded inputs and unstable systems: normalized gradient flow
Parameter estimation in the presence of dynamics: stable and unstable cases

Thu Sep 29

No class: Allerton Conference

Tue Oct 4

Online parameter estimation in SISO linear time-invariant systems
Linear parametrization (frequency domain)
Reduction to linear prediction using regressors
Gradient descent equations and preview of convergence and adaptation guarantees

Thu Oct 6

Online parameter estimation in SISO LTI systems, Part 2
Proofs of boundedness of parameter estimates, slow adaptation
Definition of persistency of excitation (PE) for vector-valued signals
Proof of exponential convergence of parameter estimates via Uniform Complete Observability and PE

Tue Oct 11

Online parameter estimation in SISO LTI systems, Part 3
Online regularized least squares: definition, motivation
Derivation of the ODE for the solution path
Proof of boundedness and convergence, role of PE

Tue Oct 13

Sufficiently rich signals and MRAC
Conditions on input to guarantee persistency of excitation: sums of sinusoidal inputs with enough distinct frequencies are sufficiently rich
Drawbacks of the PE condition: conflict with control objective
Introduction to Model Reference Adaptive Control (MRAC)

Thu Oct 18

Model Reference Adaptive Control, Part 2
Direct and indirect MRAC
Analysis of convergence via Lyapunov functions
Pitfalls of indirect MRAC: loss of stabilizability, remedy via projections

Thu Oct 20

Optimal control
Problem formulation
Dynamic programming: Bellman function, key inequalities, verification lemma
Hamilton-Jacobi-Bellman equation and existence of smooth optimal feedback controls
Linear Quadratic Regulator (LQR) problem: optimal control via Riccati Differential Equation

Tue Oct 25
[🗒 scribbles]
[🎥 video]

The LQR problem: finite and infinite-time
Finite-time: reduction to Riccati Differential Equation
Infinite-time: formulation for LTI systems and time-invariant costs
Role of controllability
Optimal solution via time reversal and Algebraic Riccati Equation

Tue Oct 27
[🗒 scribbles]
[🎥 video]

Infinite-time optimal control
LQR wrap-up (optimal control, value function, stabilizing property)
Infinite-time nonlinear stabilization: formulation, HJB equation, Bellman-Lyapunov functions

Tue Nov 1

Inifinite-time nonlinear stabilization (cont.): control-affine systems, Lie derivatives, LQR as a special case
Introduction to reinforcement learning for continuous-time deterministic systems
Infinite-time discounted cost: Bellman function, HJB equation
Value iteration and policy iteration

Thu Nov 3

Reinforcement learning in continuous time (cont.)
Value and policy updates
TD learning: motivation, implementation as gradient flow
Policy update: HJB operator, example of control-affine system with costs quadratic in control

Tue Nov 8

No class: election day.

Thu Nov 10

Optimal control revisited: the Maximum Principle
Reduction to the Mayer problem
Co-state, the adjoint equation, the Hamiltonian
Statement of the Maximum Principle for problems without terminal constraints
Relation to Dynamic Programming

Tue Nov 15

The Maximum Principle (cont.)
Co-state as the gradient of the terminal cost w.r.t. the intermediate states
Derivation of the adjoint equation
Sensitivity of control systems to parameter preturbations
Gradient computation in neural ODEs