Gradient Descent#
Gradient descent is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. It is the workhorse behind the training of most machine learning models: linear regression, logistic regression and deep neural networks are all, in practice, fitted by some variant of it.
This chapter first builds the calculus vocabulary needed to understand the algorithm — gradient vectors and directional derivatives — and then proves the statement that gives gradient descent its name: the gradient points in the direction of steepest ascent, so stepping against it decreases the function fastest. A from-scratch implementation closes the chapter.
Table of Contents#
References and Further Readings#
Zhang, Aston, Zachary C. Lipton, Mu Li, and Alexander J. Smola. “Chapter 12.3. Gradient Descent.” In Dive into Deep Learning, 2021.