The schedule will be updated and revised as the course progresses. Links to course materials will be indicated on the left.

- Tue Aug 24
- Introduction and administrivia

Examples of optimization problems

Vector spaces: axioms, examples - Thu Aug 26

[week 1 notes] - Normed spaces

Convergence, Cauchy sequences, completeness,

Banach spaces: definition, examples - Tue Aug 31
- Banach spaces, continued

Continuous functions with sup norm

Not all normed spaces are Banach spaces: continuous functions with 1-norm

Open and closed sets - Thu Sep 2

[week 2 notes] - The Banach fixed-point theorem

Contractive operators, fixed points, iteration method

Example: gradient descent - Tue Sep 7
- Continuous maps between normed spaces

Continuity vs. sequential continuity

Relatively compact and compact sets

Compactness in finite-dimensional spaces

The diagonalization argument - Thu Sep 9

[week 3 notes] - Criteria for relative compactness: the Arzelà-Ascoli theorem, total boundedness

Upper and lower semicontinuity

Optimization of functionals over compact sets: the Weierstrass theorem

Equivalence of norms in finite dimensions

Infinite-dimensional counterexamples

- Tue Sep 14
- Inner product spaces

Geometry of inner product spaces: the Pythagorean theorem and the parallelogram law

Cauchy-Schwarz inequality

Hilbert spaces: definition, key examples

Variational principles: minimum distance to a closed convex set, the projection lemma

- Thu Sep 16

[week 4 notes] - Dual space, bounded linear functionals

The Riesz representation theorem

Dimension, separability, orthonormal bases

Abstract Fourier series

The Gram-Schmidt orthogonalization procedure

Unitary equivalence of Hilbert spaces - Tue Sep 21
- Minimum norm and approximation problems in Hilbert spaces

Minimum distance to a hyperplane

Minimum-norm solutions of overdetermined linear systems: the adjoint and the pseudoinverse

Best approximation in a subspace: normal equations, Gram matrix and determinant

Finding the best approximation by Gram-Schmidt and orthogonal projection

Best approximation in closed convex hulls: the Maurey lemma, application to neural net approximation - Thu Sep 23

[week 5 notes] - Constructive Maurey: greedy selection using the Jones procedure

Minimum-norm finite-time control problem with fixed endpoint for time-varying linear systems

Controllability and observability Gramians: definition and operator-theoretic interpretation

Solution of the minimum-norm control problem using the pseudoinverse - Tue Sep 28
- Reproducing kernel Hilbert spaces

Motivation and axiomatic definition

The reproducing kernel property

Examples: finite-dimensional Hilbert spaces, bandlimited functions, Gaussian kernel

The Kolmogorov-Aronszajn construction of a unique RKHS from a symmetric, positive definite kernel

Minimum-norm problems in an RKHS - Thu Sep 30

[week 6 notes] - Hilbert spaces associated to random vectors and second-order random processes

Mean-square continuity and covariance functions

Mercer's theorem, Karhunen-Loève expansion

Example: standard Wiener process on a finite interval

- Tue Oct 5
- Linear functionals on vector spaces: algebraic dual

Continuous linear functionals on normed spaces: dual space

Banach space structure of the dual space

Examples and counterexamples of dual spaces

The extension form of the Hahn-Banach theorem

Hyperplanes, half-spaces, separation of sets - Thu Oct 7

[week 7 notes] - Separation theorems

Separating a point from a subspace: separating hyperplane

Separating a point from a closed convex set: Minkowski functional - Tue Oct 12
- The dual of
*C*[*a*,*b*]

Functions of bounded variation, Riemann-Stieltjes integral

The Riesz-Markov theorem

Minimum-norm problems: primal and dual forms

Existence of primal and dual solutions - Thu Oct 14

[week 8 notes] - Application to the moment problem on an interval

Existence of solutions for the truncated moment problem

Necessary and sufficient conditions for existence of solutions for the full moment problem

The Chebyshev approximation problem

Characterization of the dual

Structural results for continuous linear functionals on*C*[*a*,*b*]: positive linear functionals, the Jordan decomposition - Tue Oct 19
- The Chebyshev approximation problem, revisited

Structure of the dual, discrete support, probabilistic representation

The optimal control of rockets: formulation as a minimum norm problem

Relaxed controls as continuous linear functionals - Thu Oct 21

[week 9 notes] - Applications of duality in minimum-norm problems

Optimal control of rockets

Minimum-norm problems subject to multiple linear constraints: the primal and the dual formulations

Example: optimal control of a motor subject to maximum torque constraint

- Tue Oct 26
- Convex sets, supporting hyperplanes

Separation theorems

Minimum distance from a convex set in a normed space: primal and dual formulations

Global theory of convex optimization: introduction

Convex sets, convex functionals, characterization via epigraphs - Thu Oct 28

[week 10 notes] - The Legendre-Fenchel dual functional and its geometric interpretation

Constrained convex minimization problems: the Fenchel duality theorem - Tue Nov 2

[async.] - Applications of the Fenchel duality theorem

Optimal allocation of assets

Minimum-norm control in finite time

- Thu Nov 4

[week 11 notes] - Optimal transportation

Couplings and expected transportation cost

The Kantorovich formulation as a relaxed version of the Monge problem

The Kantorovich duality theorem - Tue Nov 9
- The Kantorovich duality theorem, continued

Optimal transportation problem for boudned and metric costs: the Rubinstein duality formula

Application of convex optimization to statistical hypothesis testing - Thu Nov 11
- No class
- Tue Nov 16

[week 12 notes] - Application of convex optimization to hypothesis testing, continued

The structure of optimal tests for simple hypotheses

Extension to composite hypotheses via separation of convex sets

Convex relaxation via detector-based tests (following Juditsky and Nemirovski)