While you are working on classification tasks in deep learning, especially with TensorFlow/Keras, it is important to choose the right loss function. Two commonly used loss functions are:
- binary_crossentropy (used for binary and multi-label classification)
- categorical_crossentropy (used for multi-class classification)
Though both functions are used for classification problems, they yield different performances even when applied to the same dataset. In this blog, we will explore the reason behind this and learn to decide which function to use.
Table of Contents
Understanding binary_crossentropy and categorical_crossentropy
What is binary_crossentropy?
binary_crossentropy is a loss function that is used when each sample belongs to one of the two classes (binary classification) or multiple non-exclusive classes (multi-label classification). Here, each class is treated differently.
The formula for binary_crossentropy is given below:
The model gives a single probability per class as output by using the sigmoid activation.
What is categorical_crossentropy?
categorical_crossentropy is a loss function that is used when each sample belongs to exactly one class out of multiple mutually exclusive classes (multi-class classification).
The formula for categorical_crossentropy is given below:
Here the model gives probability distribution over all classes as output by using a softmax activation.
- Label Representation Affects Learning
- binary_crossentropy treats each class independently. This can cause conflicting gradients when applied to multi-class classification.
- categorical_crossentropy ensures that a mutually exclusive probability distribution is learned by the model, which leads to more stable learning.
- Activation Function Differences
- binary_crossentropy: It uses a sigmoid activation which helps to independent probabilities.
- categorical_crossentropy: It uses a softmax activation which helps to ensure that the sum of probabilities = 1.
- Training efficiency is improved by Softmax because it forces the model to focus on learning relative class probabilities rather than treating them separately.
- Computational Stability and Performance
- categorical_crossentropy: It tends to converge faster. This is because softmax helps to distribute probability mass efficiently across multiple classes.
- Binary_crossentropy: After it is applied to multiclass problems, it can cause instability since it doesn’t force competition between classes.
- When Class Imbalance Matters
- binary_crossentropy: It allows for the better handling of imbalanced datasets in multi-label cases (e.g., detection of multiple diseases in an image.)
- categorical_crossentropy: It struggles with the highly imbalanced classes. This is because softmax forces an exclusive choice.
Experiment: Comparing Both Loss Functions on the Same Dataset
Given below are the steps to compare both the loss functions on a multi-class classification problem using the Iris dataset in Keras.
Step 1: Load and Preprocess Data
Example:
Output:
Explanation:
The above code is used to load the Iris dataset. One-hot encodes the labels and splits them into training set (80%) and test set (20%) while preserving class distribution. It normalizes the features and prints the class distribution in both sets. This helps to ensure proper stratification.
Step 2: Model Using binary_crossentropy
Example:
Output:
Explanation:
The above code is used to define a neural network with a sigmoid function. This is done for multi-label classification, which compiles it using binary cross-entropy. It then trains it on the Iris dataset and evaluates its accuracy on the test set.
Step 3: Model Using categorical_crossentropy
Example:
Output:
Explanation:
The above code is used to define a neural network with a softmax activation. This is done for multi-class classification which compiles it using categorical crossentropy, trains it on the Iris dataset, and accuracy evaluation is done on the test set.
Step 4: Compare the Results
Example:
Output:
Explanation:
The above code is used to plot the accuracy curves for both the models trained with binary cross-entropy and categorical cross-entropy. This allows us to get a visual comparison of their performance over epochs.
Key Takeaways
Some of the key information regarding the two loss functions has been provided in a tabular format below:
| Comparison Factor | Binary Crossentropy | Categorical Crossentropy |
| Suitable For | Binary and Multi-label classification. | Multi-class classification |
| Activation Function | Sigmoid | Softmax |
| Probability Outputs | Independent probabilities for each class. | Sum of probabilities = 1 |
| Performance in Multi-Class | Can lead to worse results. | More stable training. |
| Handling Imbalanced Data | Better for imbalanced multi-label data. | Not ideal for imbalanced data. |
When to Use Each Loss Function?
Choosing the right loss function is important for training an effective neural network. The decision to choose between binary_crossentropy and categorical_crossentropy depends on the nature of the classification problem. Below are the steps where we’ve discussed the decision to choose which loss function to use.
- When to Use binary_crossentropy
Best for Binary and Independent Multi-Label Classification
- You can use binary cross-entropy when only two classes are involved in the classification task (binary classification).
- You can also use it for multi-label classification, where each class is treated independently.
How it works?
- The model gives independent probabilities as outputs which uses a sigmoid activation function.
- Each class is evaluated separately, which makes it suitable for cases where multiple classes can be assigned to the same output.
Example: Binary Classification (Cat vs. Dog)
Output:
Explanation:
The above code is used to define, compile, and train a simple neural network for binary classification. It used binary_crossentropy loss, and then it evaluates its accuracy based on test data.
- When to Use categorical_crossentropy
It is best for Multi-Class Classification (Mutually Exclusive Classes)
- You can use categorical cross-entropy when more than two classes are involved in the classification task, and each input belongs to exactly one class.
- The model gives a probability distribution across all classes as outputs using the softmax activation.
How does it work?
- One neuron per class is present in the final layer with softmax activation.
- Softmax helps to ensure that the sum of predicted probabilities for all classes is exactly 1. This makes the output interpretable as a probability distribution.
Example: Multi-Class Classification (Iris Dataset)
Output:
Why use Softmax and Categorical Crossentropy together?
- It ensures that you get mutually exclusive class predictions.
- It gives one probability distribution as output, which makes classification decisions more confident and interpretable.
Comparison summary
The summary of the comparison between the two loss functions is given in the tabular format:
| Feature | binary_crossentropy | categorical_crossentropy |
| Use Case | Binary & multi-label classification. | Multi-class classification. |
| Activation | sigmoid | softmax |
| Output Shape | N x c (C independent class probabilities) | N x c (1 probability per sample) |
| Probability Sum | Does not sum to 1. | Sums to 1. |
Conclusion
The difference between binary_crossentropy and categorical_crossentropy arises from the fact about the way each loss function interprets class probabilities. Each class is treated independently by binary_crossentropy, which makes it suitable for multi-label classification but makes it suboptimal for multi-class problems where there are mutually exclusive classes. Whereas, categorical_crossentropy helps to ensure that the predicted probabilities sum to 1, which enforces a clear distinction between classes. This leads to better convergence and accuracy in multi-class classification. Therefore, it is important to choose the right loss function because it helps you to achieve optimal performance. You should always consider the nature of your classification task and how the model should interpret outputs when making the decision.
FAQs:
1. When should I use binary_crossentropy instead of categorical_crossentropy?
You can use binary_crossentropy instead of categorical_crossentropy when you are performing binary classification or multi-label classification.
2. Why does categorical_crossentropy work better for multi-class classification?
This is because categorical-cross-entropy enforces a probability distribution across all classes. This helps to ensure that the model assigns higher confidence to the correct class correctly while pushing the probabilities of the other classes lower.
3. Can I use binary_crossentropy for a multi-class classification problem?
Yes, technically you can use binary_crossentropy for a multi-class classification problem. This is because it treats each class independently. It also does not ensure that the probabilities sum to 1. This can lead to poor generalization and incorrect confidence scores.
4. How does activation function choice affect loss function performance?
A sigmoid activation is used for binary_crossentropy while a softmax activation is used for categorical_crossentropy to produce a valid probability distribution.
5. What happens if I use categorical_crossentropy for a binary classification problem?
If you use categorical_crossentropy for a binary classification problem then there won’t be any issue as categorical_crossentropy can work with a binary classification problem. However it is unnecessary and computationally inefficient for binary classification, as binary_crossentropy is specifically optimized for two-class problems.