ECE 580: Lecture Schedule

Normed and Banach spaces

Tue Aug 24

Introduction and administrivia
Examples of optimization problems
Vector spaces: axioms, examples

Thu Aug 26
[week 1 notes]

Normed spaces
Convergence, Cauchy sequences, completeness,
Banach spaces: definition, examples

Tue Aug 31

Banach spaces, continued
Continuous functions with sup norm
Not all normed spaces are Banach spaces: continuous functions with 1-norm
Open and closed sets

Thu Sep 2
[week 2 notes]

The Banach fixed-point theorem
Contractive operators, fixed points, iteration method
Example: gradient descent

Tue Sep 7

Continuous maps between normed spaces
Continuity vs. sequential continuity
Relatively compact and compact sets
Compactness in finite-dimensional spaces
The diagonalization argument

Thu Sep 9
[week 3 notes]

Criteria for relative compactness: the Arzelà-Ascoli theorem, total boundedness
Upper and lower semicontinuity
Optimization of functionals over compact sets: the Weierstrass theorem
Equivalence of norms in finite dimensions
Infinite-dimensional counterexamples

Hilbert spaces

Tue Sep 14

Inner product spaces
Geometry of inner product spaces: the Pythagorean theorem and the parallelogram law
Cauchy-Schwarz inequality
Hilbert spaces: definition, key examples
Variational principles: minimum distance to a closed convex set, the projection lemma

Thu Sep 16
[week 4 notes]

Dual space, bounded linear functionals
The Riesz representation theorem
Dimension, separability, orthonormal bases
Abstract Fourier series
The Gram-Schmidt orthogonalization procedure
Unitary equivalence of Hilbert spaces

Tue Sep 21

Minimum norm and approximation problems in Hilbert spaces
Minimum distance to a hyperplane
Minimum-norm solutions of overdetermined linear systems: the adjoint and the pseudoinverse
Best approximation in a subspace: normal equations, Gram matrix and determinant
Finding the best approximation by Gram-Schmidt and orthogonal projection
Best approximation in closed convex hulls: the Maurey lemma, application to neural net approximation

Thu Sep 23
[week 5 notes]

Constructive Maurey: greedy selection using the Jones procedure
Minimum-norm finite-time control problem with fixed endpoint for time-varying linear systems
Controllability and observability Gramians: definition and operator-theoretic interpretation
Solution of the minimum-norm control problem using the pseudoinverse

Tue Sep 28

Reproducing kernel Hilbert spaces
Motivation and axiomatic definition
The reproducing kernel property
Examples: finite-dimensional Hilbert spaces, bandlimited functions, Gaussian kernel
The Kolmogorov-Aronszajn construction of a unique RKHS from a symmetric, positive definite kernel
Minimum-norm problems in an RKHS

Thu Sep 30
[week 6 notes]

Hilbert spaces associated to random vectors and second-order random processes
Mean-square continuity and covariance functions
Mercer's theorem, Karhunen-Loève expansion
Example: standard Wiener process on a finite interval

Duality and the Hahn-Banach theorem

Tue Oct 5

Linear functionals on vector spaces: algebraic dual
Continuous linear functionals on normed spaces: dual space
Banach space structure of the dual space
Examples and counterexamples of dual spaces
The extension form of the Hahn-Banach theorem
Hyperplanes, half-spaces, separation of sets

Thu Oct 7
[week 7 notes]

Separation theorems
Separating a point from a subspace: separating hyperplane
Separating a point from a closed convex set: Minkowski functional

Tue Oct 12

The dual of C[a,b]
Functions of bounded variation, Riemann-Stieltjes integral
The Riesz-Markov theorem
Minimum-norm problems: primal and dual forms
Existence of primal and dual solutions

Thu Oct 14
[week 8 notes]

Application to the moment problem on an interval
Existence of solutions for the truncated moment problem
Necessary and sufficient conditions for existence of solutions for the full moment problem
The Chebyshev approximation problem
Characterization of the dual
Structural results for continuous linear functionals on C[a,b]: positive linear functionals, the Jordan decomposition

Tue Oct 19

The Chebyshev approximation problem, revisited
Structure of the dual, discrete support, probabilistic representation
The optimal control of rockets: formulation as a minimum norm problem
Relaxed controls as continuous linear functionals

Thu Oct 21
[week 9 notes]

Applications of duality in minimum-norm problems
Optimal control of rockets
Minimum-norm problems subject to multiple linear constraints: the primal and the dual formulations
Example: optimal control of a motor subject to maximum torque constraint

Convexity

Tue Oct 26

Convex sets, supporting hyperplanes
Separation theorems
Minimum distance from a convex set in a normed space: primal and dual formulations
Global theory of convex optimization: introduction
Convex sets, convex functionals, characterization via epigraphs

Thu Oct 28
[week 10 notes]

The Legendre-Fenchel dual functional and its geometric interpretation
Constrained convex minimization problems: the Fenchel duality theorem

Tue Nov 2
[async.]

Applications of the Fenchel duality theorem
Optimal allocation of assets
Minimum-norm control in finite time

Thu Nov 4
[week 11 notes]

Optimal transportation
Couplings and expected transportation cost
The Kantorovich formulation as a relaxed version of the Monge problem
The Kantorovich duality theorem

Tue Nov 9

The Kantorovich duality theorem, continued
Optimal transportation problem for boudned and metric costs: the Rubinstein duality formula
Application of convex optimization to statistical hypothesis testing

Thu Nov 11

No class

Tue Nov 16
[week 12 notes]

Application of convex optimization to hypothesis testing, continued
The structure of optimal tests for simple hypotheses
Extension to composite hypotheses via separation of convex sets
Convex relaxation via detector-based tests (following Juditsky and Nemirovski)