535520 Optimization Algorithms (最佳化演算法)

Instructor: Ping-Chun Hsieh
Email: pinghsieh [AT] nycu [DOT] edu [DOT] tw
References:
- Léon Bottou, Frank E. Curtis, and Jorge Nocedal, “Optimization Methods for Large-Scale Machine Learning,” 2018. (Available at https://arxiv.org/abs/1606.04838)
- Jorge Nocedal and Stephen Wright, “Numerical optimization,” 2006
- Arkadi Nemirovski and David Yudin, “Problem Complexity and Method Efficiency in Optimization,” John Wiley, 1983.
- Amir Beck, “Introduction to Nonlinear Optimization: Theory, algorithms, and applications with MATLAB,” Society for Industrial and Applied Mathematics, 2014.
- Dimitri Bertsekas, “Nonlinear Programming,” Athena Scientific, 2nd edition, 1999.
- Tor Lattimore and Csaba Szepesvari, “Bandit Algorithms,” 2019. (Available at https://tor-lattimore.com/downloads/book/book.pdf)
- Stephen Boyd and Lieven Vandenberghe, “Convex Optimization,” Cambridge University Press, 2004.

Grading
- Assignments: 45%
- Team Project: 55% (Report: 20%; Video: 15%; Review: 10%; Lightning Talk: 10%)

Week	Lecture	Date	Topics	Lecture Slides
1	1	9/2	Fundamentals: Formulations, Optimality Conditions, and Subgradients	Lec1, Lec1 annotated
2	2	9/9	Constrained Optimization and Duality	Lec2, Lec2 annotated
3	3	9/16	Constrained Optimization and Duality	Lec3, Lec3 annotated
4	4	9/23	Gradient Descent	Lec4, Lec4 annotated
5	5	10/7	Accelerated Gradient Methods	Lec5, Lec5 annotated
6	6	10/14	Stochastic Gradient Descent	Lec6, Lec6 annotated
7	7	10/21	Stochastic Gradient Descent and Variance Reduction	Lec7, Lec7 annotated
8	8	10/28	Variance Reduction and Gradient Methods for Constrained Optimization	Lec8, Lec8 annotated
9	9	11/11	Frank-Wolfe Method and Mirror Descent	Lec9, Lec9 annotated
10	10	11/18	Mirror Descent	Lec10, Lec10 annotated
11	11	11/25	Mirror Descent and Newton's Method	Lec11, Lec11 annotated
12	12	12/2	Quasi-Newton Method and Dual Ascent	Lec12, Lec12 annotated
13	13	12/9	Dual Ascent and ADMM	Lec13, Lec13 annotated
14	14	12/16	Optimizers for Neural Networks	Lec14, Lec14 annotated