ECE 580: Lecture Schedule

The schedule will be updated and revised as the course progresses. Links to course materials will be indicated on the left.

Normed and Banach spaces

Tue Aug 24
Introduction and administrivia
Examples of optimization problems
Vector spaces: axioms, examples
Thu Aug 26
[week 1 notes]
Normed spaces
Convergence, Cauchy sequences, completeness,
Banach spaces: definition, examples
Tue Aug 31
Banach spaces, continued
Continuous functions with sup norm
Not all normed spaces are Banach spaces: continuous functions with 1-norm
Open and closed sets
Thu Sep 2
[week 2 notes]
The Banach fixed-point theorem
Contractive operators, fixed points, iteration method
Example: gradient descent
Tue Sep 7
Continuous maps between normed spaces
Continuity vs. sequential continuity
Relatively compact and compact sets
Compactness in finite-dimensional spaces
The diagonalization argument
Thu Sep 9
[week 3 notes]
Criteria for relative compactness: the Arzelà-Ascoli theorem, total boundedness
Upper and lower semicontinuity
Optimization of functionals over compact sets: the Weierstrass theorem
Equivalence of norms in finite dimensions
Infinite-dimensional counterexamples

Hilbert spaces

Tue Sep 14
Inner product spaces
Geometry of inner product spaces: the Pythagorean theorem and the parallelogram law
Cauchy-Schwarz inequality
Hilbert spaces: definition, key examples
Variational principles: minimum distance to a closed convex set, the projection lemma
Thu Sep 16
[week 4 notes]
Dual space, bounded linear functionals
The Riesz representation theorem
Dimension, separability, orthonormal bases
Abstract Fourier series
The Gram-Schmidt orthogonalization procedure
Unitary equivalence of Hilbert spaces
Tue Sep 21
Minimum norm and approximation problems in Hilbert spaces
Minimum distance to a hyperplane
Minimum-norm solutions of overdetermined linear systems: the adjoint and the pseudoinverse
Best approximation in a subspace: normal equations, Gram matrix and determinant
Finding the best approximation by Gram-Schmidt and orthogonal projection
Best approximation in closed convex hulls: the Maurey lemma, application to neural net approximation
Thu Sep 23
[week 5 notes]
Constructive Maurey: greedy selection using the Jones procedure
Minimum-norm finite-time control problem with fixed endpoint for time-varying linear systems
Controllability and observability Gramians: definition and operator-theoretic interpretation
Solution of the minimum-norm control problem using the pseudoinverse
Tue Sep 28
Reproducing kernel Hilbert spaces
Motivation and axiomatic definition
The reproducing kernel property
Examples: finite-dimensional Hilbert spaces, bandlimited functions, Gaussian kernel
The Kolmogorov-Aronszajn construction of a unique RKHS from a symmetric, positive definite kernel
Minimum-norm problems in an RKHS
Thu Sep 30
[week 6 notes]
Hilbert spaces associated to random vectors and second-order random processes
Mean-square continuity and covariance functions
Mercer's theorem, Karhunen-Loève expansion
Example: standard Wiener process on a finite interval

Duality and the Hahn-Banach theorem

Tue Oct 5
Linear functionals on vector spaces: algebraic dual
Continuous linear functionals on normed spaces: dual space
Banach space structure of the dual space
Examples and counterexamples of dual spaces
The extension form of the Hahn-Banach theorem
Hyperplanes, half-spaces, separation of sets
Thu Oct 7
[week 7 notes]
Separation theorems
Separating a point from a subspace: separating hyperplane
Separating a point from a closed convex set: Minkowski functional
Tue Oct 12
The dual of C[a,b]
Functions of bounded variation, Riemann-Stieltjes integral
The Riesz-Markov theorem
Minimum-norm problems: primal and dual forms
Existence of primal and dual solutions
Thu Oct 14
[week 8 notes]
Application to the moment problem on an interval
Existence of solutions for the truncated moment problem
Necessary and sufficient conditions for existence of solutions for the full moment problem
The Chebyshev approximation problem
Characterization of the dual
Structural results for continuous linear functionals on C[a,b]: positive linear functionals, the Jordan decomposition
Tue Oct 19
The Chebyshev approximation problem, revisited
Structure of the dual, discrete support, probabilistic representation
The optimal control of rockets: formulation as a minimum norm problem
Relaxed controls as continuous linear functionals
Thu Oct 21
[week 9 notes]
Applications of duality in minimum-norm problems
Optimal control of rockets
Minimum-norm problems subject to multiple linear constraints: the primal and the dual formulations
Example: optimal control of a motor subject to maximum torque constraint


Tue Oct 26
Convex sets, supporting hyperplanes
Separation theorems
Minimum distance from a convex set in a normed space: primal and dual formulations
Global theory of convex optimization: introduction
Convex sets, convex functionals, characterization via epigraphs
Thu Oct 28
[week 10 notes]
The Legendre-Fenchel dual functional and its geometric interpretation
Constrained convex minimization problems: the Fenchel duality theorem
Tue Nov 2
Applications of the Fenchel duality theorem
Optimal allocation of assets
Minimum-norm control in finite time
Thu Nov 4
[week 11 notes]
Optimal transportation
Couplings and expected transportation cost
The Kantorovich formulation as a relaxed version of the Monge problem
The Kantorovich duality theorem
Tue Nov 9
The Kantorovich duality theorem, continued
Optimal transportation problem for boudned and metric costs: the Rubinstein duality formula
Application of convex optimization to statistical hypothesis testing
Thu Nov 11
No class
Tue Nov 16
[week 12 notes]
Application of convex optimization to hypothesis testing, continued
The structure of optimal tests for simple hypotheses
Extension to composite hypotheses via separation of convex sets
Convex relaxation via detector-based tests (following Juditsky and Nemirovski)