Loss Functions#
Learning is impossible without a way to measure how wrong a hypothesis is. A loss function assigns to every prediction a non-negative penalty, and the machinery of empirical risk minimization — and therefore of training itself — consists of minimizing its average over data.
This chapter first defines loss functions formally, including the view of a loss as a random variable, which is what licenses talking about the expected risk. It then studies two workhorses rooted in information theory: entropy together with information gain, and the cross-entropy loss.
Table of Contents#
References and Further Readings#
Jung, Alexander. “Chapter 2.3. The Loss.” In Machine Learning: The Basics. Springer Nature Singapore, 2023.