azure machine learning workbench - High accuracy but very low confidence score

Question

I am new to Machine Learning, so please have that in mind before answering. I came across challenges trying to train a neural network in workbench using CNTK with ResNet model. I followed this tutorial provided by azure [1] https://docs.microsoft.com/en-us/azure/machine-learning/desktop-workbench/scenario-image-classification-using-cntk

My first dataset a subset from ImageNet consisting of 900 images with 4 different classes car, bus, van, and truck. Afterwards, I used a subset of the dataset provided from the link underneath. [2] http://podoce.dinf.usherbrooke.ca/challenge/dataset/

I used 9000 images of the dataset divided equally into four different into the same classes as with ImageNet and started training my network.

The classifier I used for this was the DNN classifier with the following configuration:

 rf_pretrainedModelFilename = "ResNet_50.model" 

 rf_inputResoluton = 224                

 rf_dropoutRate    = 0.5                 

 rf_mbSize         = 10               

 rf_maxEpochs      = 30                

 rf_maxTrainImages = float('inf')        

 rf_lrPerMb        = [0.01] * 10 + [0.001] * 10 + [0.0001] 

 rf_momentumPerMb  = 0.9                 

 rf_l2RegWeight    = 0.0005              

 rf_boFreezeWeights      = False         

 rf_boBalanceTrainingSet = False          images

After training the model I got an overall accuracy of 96.80% with all classes having an accuracy > 92 %. All well and done, but when I tested various other test images, my confidence score was 12.9895 at its highest peak. I got a JSON object returned like this: Image classified as 'Bus' with confidence score 12.9895.

  {\"score\": \"12.9895\", \"Id2Labels\": \"{0: 'Bus', 1: 'Truck', 2: ' 

  Car', 3: 'Van'}\", \"label\": \"Bus\", \"executionTimeMs\": \"128.749\", 

  \"allScores\": \"[ 12.98949814   3.51014233  -6.96435881  -6.89878178]\"}"

The value 12.9895 must mean 12.9895% possibility for the image being a bus, right? and why is it not returned as a value between 0 and 1? Please correct me if I am wrong, as I do get confused over the various terms being used in Machine Learning for the same thing.

Why are the minus values there, I thought the activation function took care of the minus values?

Should I include an even larger dataset or maybe better image quality to improve my score?

Any other suggestions on how I can improve my score?

The score was low on both datasets mentioned, (Subset from ImageNet and MIO). A humble thank you, for taking the time answering these questions.

Answer 1

You can use softmax function to convert the values into probability. Also, scoring means the same as prediction. The value indicates the prediction of your model when you have given some new data to your trained data. But, it can represent a category or outcome. Click here to read more.

Use Rectified Linear Unit (ReLu) function to set the negative values to zero and others remain as it is. 

Set a specified window for your images to read and identify the image as bus or truck. And larger the data set, the better.