Advanced topics in optimization: From simple to complex ML systems

Email (instructor): anastasios@rice.edu	Web: https://akyrillidis.github.io/comp545/
	Email (course): RiceCOMP545@gmail.com
Office hours: By appointment	Class hours: T\TH 15:10 - 16:30
Office: DH 3119	Classroom: Online

Course Syllabus

LaTEX template for scribing

Course description

Schedule

Grading policy

Literature

Lecture 1.
Textbook:	(None)
Other references:	- Proximal policy optimization algorithms
	- OpenAI Five
	- AlphaStar: Mastering the Real-Time Strategy Game StarCraft II
	- Semantic Image Synthesis with Spatially-Adaptive Normalization
	- GauGAN: Changing Sketches into Photorealistic Masterpieces
	- Combinatorial optimization by simulating adiabatic bifurcations in nonlinear Hamiltonian systems
	- Quantum supremacy using a programmable superconducting processor
	- On "Quantum" supremacy
	- Physicists in China challenge Google's 'quantum advantage'

---

---

Lecture 2.
Textbook:	(None)
Other references:	- Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
	- RMSProp algorithm
	- Adam: A method for stochastic optimization
	- On the convergence of adam and beyond
	- Decaying momentum helps neural network training
	- Quasi-hyperbolic momentum and Adam for deep learning
	- Aggregated Momentum: Stability Through Passive Damping
	- An overview of gradient descent optimization algorithms
	- AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
	- ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
	- AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
	- On the Variance of the Adaptive Learning Rate and Beyond

</table>

---

Lecture 3.
Textbook:	- Convex optimization (Chapter 5), Stephen Boyd and Lieven Vandenberghe
	- Algorithms for optimization (Chapter 10), Mykel Kochenderfer and Time Wheeler
	- Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein
Other references:
	- Differentiable Implicit Layers
	- Deep Declarative Networks: A New Hope
	- Differentiation of Blackbox Combinatorial Solvers
	- SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver
	- Differentiable Convex Optimization Layers
	- Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling (and references therein)
	- OptNet: Differentiable Optimization as a Layer in Neural Networks

Lecture 4.
Textbook:	- Convex optimization (Chapter 9.5-9.7, Chapter 11), Stephen Boyd and Lieven Vandenberghe
	- A mini-course on convex optimization (Chapter 3), Nisheeth Vishnoi
	- Convex Optimization: Algorithms and Complexity (Chapter 5.3), Sebastien Bubeck.
Other references:	- Faster Convex Optimization: Simulated Annealing with an Efficient Universal Barrier
	- Solving Tall Dense SDPs in the Current Matrix Multiplication Time (especially the references therein for a background on TCS + IPM)

---

Lecture 5.
Textbook:	- A mini-course on convex optimization (Chapter 2), Nisheeth Vishnoi
	- Convex Optimization: Algorithms and Complexity (Chapter 4), Sebastien Bubeck.
Other references:	- The Multiplicative Weights Update Method: a Meta Algorithm and Applications

---

Lecture 6.
Textbook:	(None)
Other references:	- Adversarial Robustness - Theory and Practice

---