Why is binary_crossentropy more accurate than categorical_crossentropy for multiclass classification in Keras?

Question

I'm learning how to create convolutional neural networks using Keras. I'm trying to get a high accuracy for the MNIST dataset.

Apparently categorical_crossentropy is for more than 2 classes and binary_crossentropy is for 2 classes. Since there are 10 digits, I should be using categorical_crossentropy. However, after training and testing dozens of models, binary_crossentropy consistently outperforms categorical_crossentropy significantly.

On Kaggle, I got 99+% accuracy using binary_crossentropy and 10 epochs. Meanwhile, I can't get above 97% using categorical_crossentropy, even using 30 epochs (which isn't much, but I don't have a GPU, so training takes forever).

Here's what my model looks like now:

model = Sequential()

model.add(Convolution2D(100, 5, 5, border_mode='valid', input_shape=(28, 28, 1), init='glorot_uniform', activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(100, 3, 3, init='glorot_uniform', activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Dropout(0.3))

model.add(Flatten())

model.add(Dense(100, init='glorot_uniform', activation='relu'))

model.add(Dropout(0.3))

model.add(Dense(100, init='glorot_uniform', activation='relu'))

model.add(Dropout(0.3))

model.add(Dense(10, init='glorot_uniform', activation='softmax'))

model.compile(loss='binary_crossentropy', optimizer='adamax', metrics=['accuracy'])

Answer 1

You should simply try to calculate the accuracy manually, and you will see that it is different from the one reported by Keras with the model.evaluate method.

For example:

score = model.evaluate(x_test, y_test, verbose=0) 

score[1]

# 0.99794011611938471

# Actual accuracy calculated manually:

import numpy as np

y_pred = model.predict(x_test)

acc = sum([np.argmax(y_test[i])==np.argmax(y_pred[i]) for i in range(10000)])/10000

acc

# 0.98999999999999999

Hope this answer helps you! For more details, go through Machine Learning Online Course. Also, study the Tensorflow Tutorial for more knowledge.