Why is the Cross Entropy method preferred over Mean Squared Error? In what cases does this doesn't hold up?

Question

Although both of the above methods provide a better score for the better closeness of prediction, still cross-entropy is preferred. Is it in every case or there are some peculiar scenarios where we prefer cross-entropy over MSE?

Answer 1

Cross-entropy loss, or log loss, measure the performance of a classification model whose output is a probability value between 0 and 1. It is preferred for classification, while mean squared error (MSE) is one of the best choices for regression. This comes directly from the statement of your problems itself. In classification you work with a very particular set of possible output values thus MSE is badly defined.

To better understand the phenomena it is good to follow and understand the relations between

1. Cross-entropy

2. Logistic regression (binary cross-entropy)

3. Linear regression (MSE)

You will notice that both can be seen as a maximum likelihood estimator (MLE), simply with different assumptions about the dependent variable.

When you derive the cost function from the aspect of probability and distribution, you can observe that MSE happens when you assume the error follows Normal Distribution and cross-entropy when you assume binomial distribution. It means that implicitly when you use MSE, you are doing regression (estimation) and when you use CE, you are doing classification.

I hope it helps a little bit.

Answer 2

Cross-Entropy (log loss) is used in classification tasks mainly when the model outputs probabilities, for instance by using sigmoid or SoftMax. It compares best with the true class labels as predicted probabilities and assumes alignment either with binomial (binary) or multinomial (multi-class) distributions. It is preferred over Mean Squared Error in classification since it maximizes the probability of the categorical output by adjusting for the correct class.

Mean Squared Error (MSE) is used when working on regression because we are predicting continuous values — lets average out the squares of the differences between the actual and predicted values. MSE assumes that errors follow a normal distribution. It is apt for regression but not for classification, classification outputs categorical data.

Both are Maximum Likelihood Estimators (MLE), but:

Cross-Entropy is based on the binomial distribution (classification).

MSE is based on the normal distribution (regression).

In summary, Cross-Entropy is for classification (probabilistic outputs) and MSE is for regression (continuous outputs).