Epoch vs Iteration when training neural networks?

Question

What is epoch and iteration and what's the difference between them while training a multi-layer perceptron .

Answer 1

An epoch is defined as the number of times an algorithm visits the data set .In other words, epoch is one backward and one forward pass for all the training.

Iteration is defined as the number of times a batch of data has passed through the algorithm.In other words, it is the number of passes, one pass consists of one forward and one backward pass.

For example-

Consider a set of 1000 images having a batch size of 10 and each iteration would process 10 images, it will take100 such iterations to complete 1 epoch.  

For more insights visit this Neural Network Tutorial.

Answer 2

You have a fixed training set of your neural network for building a model. The role of an epoch is to train your network on each item of your model i.e. one epoch is one forward pass and one backward pass of all the training examples.

So, if you want to teach your neural network to recognize the letters of the alphabet, 100 epochs would mean you have 2600 individual training trials

.

For 26 alphabets

for(i=0;i<epochs;i++)

{

for(j=1;j<=26;j++)

{

//train each alphabet i.e update the weights of neural network;

}

}

So, what are the right numbers of epochs?

We cannot predict the right numbers of epochs because it is different for different datasets but yes you can tell the number of epochs by looking at the diversity of your data.

For example: Do you have only black cats in your dataset or is it a much more diverse dataset?

If your data contains only black cats then the epochs will be constant and if your data is more diversified then your number of epochs will vary.

batch size is the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.

Whereas, an iteration indicates the number of times your algorithm’s parameters are updated and depends upon the context. A single iteration involves the following steps:

1. Processing the training dataset batch.

2. Calculating the cost function.

3. Backpropagation and adjustment of all the weights.

Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.

Watch this video to learn about Neural Networks: