CS 531: Advanced Optimization Algorithms

Course Description

This is a graduate-level course on the design and analysis of iterative algorithms for continuous and discrete optimization. Iterative methods based on gradient descent have had a profound impact on many areas of computer science and engineering, including algorithms and theoretical computer science, machine learning, data mining, finance, and operations research. The main topics covered are the design and analysis of gradient descent methods; adaptive and stochastic gradient descent; linear programming and duality; online learning and multiplicative weight update method; and how these frameworks can be used to obtain very efficient algorithms for classification, regression, discrete optimization, and beyond.

Syllabus

Collaboration and Honesty Policy

Piazza

Gradescope for submitting homeworks. Sign up using the entry code 68ZEXX.

Course Staff

Prof. Alina Ene
Homepage: cs-people.bu.edu/aene/
Email: aene @ bu . edu
Office Hours: Mondays 11:30am - 12:30pm, Tuesdays 10:45am - 11:45am, Thursdays 10:45am - 11:45am (in CDS 1027)

Teaching Fellow: Duy Le
Email: duyle @ bu . edu
Office Hours: Fridays noon - 2pm (in CDS 10th floor lounge)

Lectures

Tue/Thu 9:30am - 10:45am, in PSY B35

The schedule is tentative and subject to change (e.g., snow days).

Lecture	Topic
Mathematical background. Introduction to optimization.
9/3	Course overview and introduction. Linear classification and the Perceptron algorithm.
9/8, 9/10	Review of concepts from linear algebra and multivariate calculus.
9/15, 9/17	Introduction to optimization, examples of optimization problems.
9/22	Optimality conditions for general problems.
Convex optimization. Gradient descent algorithms.
9/24	Convex functions and sets, optimality conditions for convex problems.
9/29	Oracle models, iterative methods, and gradient descent.
10/1, 10/6	Gradient descent algorithms for convex optimization problems.
Supervised learning. Stochastic and adaptive gradient descent.
10/8	Supervised learning. Linear models. Algorithms for linear regression.
10/13	No class (Monday schedule)
10/15	Algorithms for linear classification.
10/20, 10/22	Neural network models. Stochastic gradient descent.
10/27	Adaptive gradient descent algorithms.
10/29	In-class midterm exam
Linear programming. LP duality. Algorithmic frameworks based on LPs and duality.
11/3	Introduction to linear programming. Modeling using LPs.
11/5	LP duality.
11/10, 11/12	Algorithmic frameworks based on LPs and duality.
11/17, 11/19	Duality applications.
Optimization under uncertainty: online learning, online algorithms
11/24	Prediction using expert advice. Majority algorithms.
11/26	No class (Thanksgiving break)
12/1	Multiplicative weights update algorithm. Application to classification: Winnow algorithm.
12/3	Multiplicative weights update algorithm applications: solving positive LPs.
12/8, 12/10	Online algorithms: sky rental, caching, linked list maintenance.

Acknowledgments: I am indebted to my colleagues at other institutions for some of the material in the lectures: Amir Ali Ahmadi's course at Princeton, Yaron Singer's course at Harvard, Nick Harvey's course at UBC, ... . The specific references/credits are on the References slide at the end of each lecture.