535526 Fall 2022 - Optimization Algorithms (最佳化演算法)

Instructor: Ping-Chun Hsieh
Email: pinghsieh [AT] nycu [DOT] edu [DOT] tw
Lectures: Mondays 12:20pm-3:10pm @ EC015
Office Hours: 3:10pm-4pm on Mondays or by appointment
References:
- Léon Bottou, Frank E. Curtis, and Jorge Nocedal, “Optimization Methods for Large-Scale Machine Learning,” 2018. (Available at https://arxiv.org/abs/1606.04838)
- Jorge Nocedal and Stephen Wright, “Numerical optimization,” 2006
- Arkadi Nemirovski and David Yudin, “Problem Complexity and Method Efficiency in Optimization,” John Wiley, 1983.
- Amir Beck, “Introduction to Nonlinear Optimization: Theory, algorithms, and applications with MATLAB,” Society for Industrial and Applied Mathematics, 2014.
- Dimitri Bertsekas, “Nonlinear Programming,” Athena Scientific, 2nd edition, 1999.
- Tor Lattimore and Csaba Szepesvari, “Bandit Algorithms,” 2019. (Available at https://tor-lattimore.com/downloads/book/book.pdf)
- Stephen Boyd and Lieven Vandenberghe, “Convex Optimization,” Cambridge University Press, 2004.

Grading
- Assignments: 45%
- Team Project: 55% (Report: 20%; Video: 15%; Review: 10%; Lightning Talk: 10%)

Week	Lecture	Date	Topics	Lecture Slides
1	1	9/12	Logistics and Fundamentals
2	2	9/19	Subgradients, Constrained Optimization, and Lagrangians
3	3	9/26	Duality
4	4	10/3	Gradient Descent (GD)
5		10/10	National Holiday
6	5	10/17	GD and SGD
7	6	10/24	SGD and Variance Reduction
8		10/31	No Class (Rescheduled to 1/9)
9	7	11/7	Variance Reduction, Projected GD, and Frank-Wolfe
10	8	11/14	Projected GD and Frank-Wolfe
11	9	11/21	Accelerated Gradient Methods
12	10	11/28	Mirror Descent
13	11	12/5	Mirror Descent and Proximal Gradient
14	12	12/12	Newton and Quasi-Newton Methods
15	13	12/19	Quasi-Newton and Block Coordinate Descent
16		12/26	Rescheduled for Lightning Talks (Final Exam Week)
17		1/2	National Holiday
17		1/7	Team Presentations (Lightning Talks)
18	14	1/9	ADMM