FAQ: The Machine Learning Framework#
Knowing the joint probability distribution of \(\mathbf{X}\) and \(Y\) means that the conditional probability of \(Y\) given \(\mathbf{X}\) is fully determined. This is evident from Bayes’ theorem:
\[\begin{split} \begin{align*} \mathbb{P}(Y=y\ |\ \mathbf{X} = \mathbf{x}) &= \frac{\mathbb{P}(\mathbf{X} = \mathbf{x}\ |\ Y=y)\ \mathbb{P}(Y=y)}{\mathbb{P}(\mathbf{X} = \mathbf{x})} \\ \end{align*} \end{split}\]since knowing the joint distribution \(\mathbb{P}_{\mathcal{D}}(\mathcal{X}, \mathcal{Y}; \boldsymbol{\theta})\) means the numerator \(\mathbb{P}(\mathbf{X} = \mathbf{x}\ |\ Y=y)\ \mathbb{P}(Y=y)\) is fully determined since they are equal (recall the numerator is none other the probability of \(\mathbf{X}\) and \(Y\) happening at the same time). Then once the joint distribution is known, the denominator \(\mathbb{P}(\mathbf{X} = \mathbf{x})\) is also fully determined since it is the marginal probability of \(\mathbf{X}\). It follows that the conditional probability of \(Y\) given \(\mathbf{X}\) is fully determined.
It is good to realize the ultimate goal in learning a supervised problem is generally specified as finding the conditional probability of \(Y\) given \(\mathbf{X}\), which is fully determined once the joint distribution is known.
PCA#
See the geometry of the multivariate Gaussian on treatment and intuition of eigenvectors and eigenvalues.
Loss/Cost Functions (Optimization)#
Error, Cost, Loss and Risk