How can I enrich a Convolutional Neural Network with meta information?

Question

I would very much like to understand how I can enrich a CNN with provided meta information. As I understand, a CNN 'just' looks at the images and classifies it into objects without looking at possibly existing meta-parameters such as time, weather conditions, etc.

To be more precise, I am using a Keras CNN with TensorFlow in the backend. I have the typical Conv2D and MaxPooling Layers and a fully connected model at the end of the pipeline. It works nicely and gives me good accuracy. However, I do have additional meta information for each image (the manufacturer of the camera with which the image was taken) that is unused so far.

What is a recommended way to incorporate this meta-information into the model? I could not yet come out with a good solution by myself.

Thanks for any help!

Answer 1

Normally, it is accomplished by adding this information to one of the fully connected layers before the prediction. The fully connected layer gives you a 'K' number of features representing your image, you just concatenate them with the additional information you have.

If you have K features representing the image + 1 meta information. This would mean your fully connected layer had K+1 input nodes, one of them being the meta-information.

Just think of the Convolutional Neural Network "before" the fully connected layers as a means to extract features of the images. Later in the fully connected layers, these features are used together for the classification.