Advanced topics in optimization: From simple to complex ML systems
| Email (instructor): anastasios@rice.edu | Web: https://akyrillidis.github.io/comp545/ |
| Email (course): RiceCOMP545@gmail.com | |
| Office hours: By appointment | Class hours: T\TH 15:10 - 16:30 |
| Office: DH 3119 | Classroom: Online |
| Course Syllabus | LaTEX template for scribing |
| Course description | Schedule | Grading policy | Literature |
| Lecture 1. | 1 session (Notes) | ||
| Includes: | Motivation: Popular science and optimization | Overview of the course | |
| Course logistics |
| Interlude. | 2 sessions (COMP414/514) | ||
| Includes: | Overview of smooth unconstrained optimization | Intro to convex optimization | |
| Gradient descent variants |
| Lecture 2. | 2 sessions + presentation day (Slides) (Notes) | ||
| Includes: | Overview of algorithms in modern neural network training | Focus on AdaGrad, RMSProp, Adam | |
| Discussion on the value of adaptive methods |
| Lecture 3. | 3 sessions + presentation day (Slides) (Notes) | ||
| Includes: | Constrained optimization, Lagrange multipliers | Dual problems, Weak-strong duality and KKT conditions | |
| Dual ascent, augmented Lagrangian, dual decomposition, ADMM |
| Lecture 4. | 3 sessions (Slides) (Notes) | ||
| Includes: | Interior point methods | Barrier methods | |
| Path following methods, complexity and convergence rate analysis overview |
| Interlude. | 1 session (Notes) | ||
| Includes: | Applications for the settings considered so far | ||
| Lecture 5. | 4 sessions (Notes) | ||
| Includes: | Majority weighted algorithm | Multiplicative weights update (MWU) algorithm | |
| Test cases: solving LPs, Winnow algorithm, Adaboost | MWU and mirror descent |
| Lecture 6. | 4 sessions (Notes) | ||
| Includes: | Adversarial robustness in ML as minmax opt. | Algorithms for adversarial examples | |
| Algorithms for adversarial defense |
| Lecture 7. | 2 sessions | ||
| Includes: | A case of discrete optimization over quadratic forms | ||
| Lecture 8. | 2 sessions | ||
| Includes: | TBD | ||
| Seminar | 2-3 sessions | ||
| Includes: | Student presentations | ||