ECE 543: Lecture Schedule
The schedule will be updated and revised as the course progresses. Required reading from the lecture notes will be indicated on the left.
Preliminaries
- Tue Jan 16
📖 Ch. 1
- Introduction and administrivia
Goals of learning
- Thu Jan 18
Tue Jan 23
📖 Ch. 2
- Concentration inequalities
- Chernoff method and subgaussian random variables
- Hoeffding's inequality
- McDiarmid's inequality, bounded differences
Basic theory
- Thu Jan 25
Tue Jan 30
📖 Ch. 3
- Formulation of the learning problem
- realizable case: concept and function learning
- Probably Approximately Correct (PAC) learning
- agnostic (model-free) learning
- consistency and uniform convergence of empirical means
- Empirical Risk Minimization
- Thu Feb 1
Tue Feb 6
📖 Ch. 4
- Empirical Risk Minimization: abstract risk bounds
- excess risk of ERM via uniform deviation
- bounding the uniform deviation via Rademacher averages, symmetrization
- structural properties of Rademacher averages, Finite Class Lemma
- Thu Feb 8
Tue Feb 13
📖 Ch. 5
- Vapnik-Chervonenkis classes
- shatter coefficients, VC dimension
- examples: intervals, half-spaces, axis-parallel rectangles, finite-dimensional function spaces
- growth of shatter coefficients: Sauer-Shelah lemma, Pajor's theorem
- Thu Feb 15
Tue Feb 20
Thu Feb 22
Tue Feb 27
📖 Ch. 6
- Binary classification
- linear discriminant rules and generalized linear discriminant rules
- risk bounds for combined classifiers via surrogate losses
- weighted linear combinations of classifiers, margin
- kernel machines, RKHS
- generalization bounds for AdaBoost via surrogate losses
- neural nets
- Thu Mar 1
📖 Ch. 7
- Regression with squared loss
- regression over a ball in RKHS
- regression over RKHS with additive regularization
Advanced topics
- Tue Mar 6
Tue Mar 13
Thu Mar 15
📖 Ch. 11
- Stability of learning algorithms
- learnability without uniform convergence
- generalization error, stability of learning algorithms
- stability of ERM under strong convexity
- stability of Stochastic Gradient Descent (SGD)
- convergence and optimality guarantees for SGD
- Thu Mar 8
- In-class review before Midterm 1
- Tue Mar 20
Thu Mar 22
- SPRING BREAK
- Tue Mar 27
Thu Mar 29
Tue Apr 3
Thu Apr 5
📖 Ch. 12
- Online learning
- online learning model: Forecaster vs. Adversary, strategies
- performance criteria: cumulative loss, comparators, regret
- regret bounds for online convex optimization: convex Lipschitz functions; strongly convex Lipschitz functions
- the online perceptron algorithm: sample complexity via regret bounds
- generalization ability of online learning algorithms: online-to-batch conversion, martingale decomposition
- Tue Apr 10
Thu Apr 12
Tue Apr 17
📖 Ch. 13
- Minimax lower bounds
- minimax excess risk in binary classification, Massart noise condition
- preliminaries: binary hypothesis testing, a bit of information theory
- minimax lower bounds of Massart and Nédélec, dependence on noise margin, rich classes
Some applications
- Thu Apr 19
📖 Ch. 8
- Empirical vector quantization
- lossy source coding: vector quantization, bit rate, expected distortion
- fixed-rate quantizers, nearest-neighbor property, optimality of nearest-neighbor quantizers
- empirically optimal quantizers
- bounds on expected distortion of empirically optimal quantizers via VC theory