ECE 490: Lecture Schedule

Tue Jan 21

No class (❄️❄️❄️)

Thu Jan 23
Ch. 1

Introduction and administrivia
Data analysis and optimization: examples

Tue Jan 28
Thu Jan 30
Ch. 2

Foundations of smooth optimization
Taylor's theorem and some consequences
Notions of smoothness: Lipschitz continuity, Lipschitz gradients
Necessary and sufficient conditions for local minima
Convexity and strong convexity, interplay with Lipschitz gradients

Tue Feb 4
Thu Feb 6
Sections 3.1, 3.2, 3.4, 3.5

Descent methods
Directions of descent: definition, sufficient conditions
Steepest descent method (a.k.a. gradient method)
Convergence rates for smooth or smooth + strongly convex functions
Line-search methods: choosing the direction and the steplength

Tue Feb 11
Sections 3.5, 3.6
Thu Feb 13
Ch. 4

Descent methods (cont.)
Approximate line-search: weak Wolfe conditions
Mirror descent
Gradient methods using momentum
Motivation using differential equations
Heavy ball method and Nesterov's accelerated method

Tue Feb 18

Review and Q&A

Thu Feb 20

In-class midterm 1

Tue Feb 25
Thu Feb 27
Sections 5.1-5.3

Stochastic gradient methods
Motivation and examples
Randomized incremental gradient
Robbins-Monro scheme
Randomized Kaczmarz method
Key assumptions on stochastic gradients, with examples

Tue Mar 4
Sections 5.4-5.5
Thu Mar 6
Sections 6.1-6.2

Stochastic gradient methods
Convergence analysis
Implementation aspects: epochs, mini-batching, momentum
Coordinate descent
Motivation and examples from machine learning
Local Lipschitz constants
Convergence analysis of randomized coordinate descent: sampling with replacement

Tue Mar 11
Thu Mar 13
Ch. 7

First-order methods for constrained optimization
Directions of descent, feasible directions, normal cone
First-order necessary condition for local optimality
Euclidean projection onto a closed convex set: definition, examples, the minimum principle
Projected gradient descent: convergence analysis
Conditional gradient (Frank-Wolfe) method

Tue Mar 18
Thu Mar 20

No class: Spring Break

Tue Mar 25
Thu Mar 27
Ch. 8

Nonsmooth functions and subgradients
Subgradients and subdifferentials: motivation, definition, examples
First-order optimality condition in terms of subgradients
A supporting hyperplane interpretation

Tue Apr 1

Review and Q&A

Thu Apr 4

In-class midterm 2

Tue Apr 8

No class

Thu Apr 10
Ch. 8

Nonsmooth functions and subgradients (cont.)
Calculus of subdifferentials (additivity, composition with affine transformations, Danskin's theorem)
Convex sets and convex constrained optimization

Tue Apr 15
Thu Apr 17
Ch. 9

Nonsmooth optimization methods
Motivating example for nonsmooth optimization: LASSO
Direction of steepest descent: minimum-norm subgradient
Subgradient descent: rates of convergence with constant and dmininishing step lengths
Proximal gradient methods for regularized optimization

Tue Apr 22
Thu Apr 24
Ch. 10

Duality and algorithms
Optimization subject to linear equality constraints
Quadratic penalty, Lagrangians, weak duality
Necessary and sufficient conditions for optimality
Strong duality
Dual optimization algorithms
Augmented Lagrangian method, ADMM
Examples

Tue Apr 29
Thu May 1
Ch. 11

Differentiation and adjoints
Multivariable chain rule, Jacobian-vector products
Computation of gradients of nested compositions of vector-valued functions
The method of adjoints and its Lagrangian interpretation
Automatic differentiation and backpropagation

Tue May 6

Review and Q&A