ECE 517: Lecture Schedule

Tue Aug 25
[scribbles]

Introduction and administrivia
Adaptation as control in the presence of large-scale uncertainty
Some history: gain scheduling, model reference adaptive control, Lyapunov redesign, self-tuning controllers, direct vs. indirect adaptive control
Basic example: adaptive output regulation of a first-order scalar linear system
Lyapunov stability (review)
Certainty equivalence principle

Thu Aug 27
[scribbles]

Stability
Weak Lyapunov functions
Krasovskii-LaSalle theorem
Barbalat's lemma
Adaptive regulation revisited

Tue Sep 1
[scribbles]

Review: weak Lyapunov functions
Connections to observability: Uniform Complete Observability (Kalman)
Adaptive regulation revisited: alternative analysis with V(x) = x²/2 using boundedness and Barbalat's lemma
Universal regulators: definition
Example: scalar linear plant dy/dt = a y + b u, known vs. unknown sign of b

Thu Sep 3
[scribbles]

Universal regulation for linear time-invariant systems
Scalar systems:

• fundamental limits (controllers with rational dynamics cannot achieve universal regulation)
• achievability of universal regulation via Nussbaum gains, exploration vs. exploitation

Tue Sep 8
[scribbles]

Universal regulation for time-invariant linear systems, continued
Higher-dimensional systems:

• preliminaries (relative degree, minimum phase)
• achievability of universal regulation for SISO systems via output feedback with Nussbaum gains

Thu Sep 10
[scribbles]

Introduction to Lyapunov-based design
Universal regulation example revisited
Control Lyapunov functions and continuous selection problem
Sontag's universal formula for control-affine systems

Tue Sep 15
[scribbles]

Sontag's universal formula, continued
Motivation through the Linear Quadratic Regulator problem
Comparison and contrast with other stabilizing controllers (e.g., feedback linearization)
Application to adaptive stabilization of a first-order scalar linear plant

Thu Sep 17
[scribbles]

Backstepping
Motivation and procedure for control-affine systems
Comparison with feedback linearization

Tue Sep 22
[scribbles]

Adaptive backstepping
Effect of unknown parameters and iterative tuning function update
Introduction to parameter estimation

Thu Sep 24
[scribbles]

Optimization in continuous time
Global minima, first-order optimality condition
Gradient flow
Convex and strongly convex functions
Lyapunov analysis

Tue Sep 29
[scribbles]

Online optimization in continuous time, Part 1
Motivation through parameter estimation in adaptive control
The online optimization paradigm: asymptotic no-regret property
No-regret in online convex optimization via gradient flow
Analysis for uniformly Lipschitz costs via Lyapunov functions
Convergence criteria for costs and parameter estimates via Barbalat

Thu Oct 1
[scribbles]

Online optimization in continuous time, Part 2
Analysis for uniformly strongly convex costs
Convergence criteria for costs and parameter estimates: persistency of excitation as integrated strong convexity

Tue Oct 6
[scribbles]

Online optimization in continuous time, Part 3
Extension to unbounded inputs and unstable systems: normalized gradient flow
Parameter estimation in the presence of dynamics: stable and unstable cases

Thu Oct 8
[scribbles]

Online parameter estimation in SISO linear time-invariant systems
Linear parametrization (frequency domain)
Reduction to linear prediction using regressors
Gradient descent equations and preview of convergence and adaptation guarantees

Tue Oct 13
[scribbles]

Online parameter estimation in SISO LTI systems, Part 2
Proofs of boundedness of parameter estimates, slow adaptation
Definition of persistency of excitation (PE) for vector-valued signals
Proof of exponential convergence of parameter estimates via Uniform Complete Observability and PE

Thu Oct 15
[scribbles]

Online parameter estimation in SISO LTI systems, Part 3
Online regularized least squares: definition, motivation
Derivation of the ODE for the solution path
Proof of boundedness and convergence, role of PE

Tue Oct 20
[scribbles]

Sufficiently rich signals and MRAC
Conditions on input to guarantee persistency of excitation: sums of sinusoidal inputs with enough distinct frequencies are sufficiently rich
Drawbacks of the PE condition: conflict with control objective
Introduction to Model Reference Adaptive Control (MRAC)

Thu Oct 22
[scribbles]

Model Reference Adaptive Control, Part 2
Direct and indirect MRAC
Analysis of convergence via Lyapunov functions
Pitfalls of indirect MRAC: loss of stabilizability, remedy via projections

Tue Oct 27
[scribbles]

Optimal control
Problem formulation
Dynamic programming: Bellman function, key inequalities, verification lemma
Hamilton-Jacobi-Bellman equation and existence of smooth optimal feedback controls
Linear Quadratic Regulator (LQR) problem: optimal control via Riccati Differential Equation

Thu Oct 29
[scribbles]

The LQR problem: finite and infinite-time
Finite-time: reduction to Riccati Differential Equation
Infinite-time: formulation for LTI systems and time-invariant costs
Role of controllability
Optimal solution via time reversal and Algebraic Riccati Equation

Tue Nov 3

No class: Presidential Election

Thu Nov 5
[scribbles]

Infinite-time optimal control
LQR wrap-up (optimal control, value function, stabilizing property)
Infinite-time nonlinear stabilization: formulation, HJB equation, Bellman-Lyapunov functions

Tue Nov 10
[scribbles]

Inifinite-time nonlinear stabilization (cont.): control-affine systems, Lie derivatives, LQR as a special case
Introduction to reinforcement learning for continuous-time deterministic systems
Infinite-time discounted cost: Bellman function, HJB equation
Value iteration and policy iteration

Thu Nov 12
[scribbles]

Reinforcement learning in continuous time (cont.)
Value and policy updates
TD learning: motivation, implementation as gradient flow
Policy update: HJB operator, example of control-affine system with costs quadratic in control

Tue Nov 17
[scribbles]

Optimal control revisited: the Maximum Principle
Reduction to the Mayer problem
Co-state, the adjoint equation, the Hamiltonian
Statement of the Maximum Principle for problems without terminal constraints
Relation to Dynamic Programming

Thu Nov 19
[scribbles]

The Maximum Principle (cont.)
Co-state as the gradient of the terminal cost w.r.t. the intermediate states
Derivation of the adjoint equation
Sensitivity of control systems to parameter preturbations
Gradient computation in neural ODEs