Neural Network / Machine Learning memory storage

Question

I am currently trying to set up a Neural Network for information extraction and I am pretty fluent with the (basic) concepts of Neural Networks, except for one which seems to puzzle me. It is probably pretty obvious but I can't seem to found information about it.

Where/How do Neural Networks store their memory? ( / Machine Learning)

There is quite a bit of information available online about Neural Networks and Machine Learning but they all seem to skip over memory storage. For example, after restarting the program, where does it find its memory to continue learning/predicting? Many examples online don't seem to 'retain' memory but I can't imagine this being 'safe' for real/big-scale deployment.

I have a difficult time wording my question, so please let me know if I need to elaborate a bit more. Thanks,

To follow up on the answers below

Every Neural Network will have edge weights associated with them. These edge weights are adjusted during the training session of a Neural Network.

This is exactly where I am struggling, how do/should I vision this secondary memory? Is this like RAM? that doesn't seem logical. The reason I ask because I haven't encountered an example online that defines or specifies this secondary memory (for example in something more concrete such as an XML file, or maybe even a huge array).

Answer 1

You should understand that memory storage is implementation-specific, it’s not the part of the algorithm. It is quite useful to examine what you need to store rather than how to store it.

For example:

Consider a 3-layer multi-layer perceptron (fully connected) that has 3, 8, and 5 nodes in the input, hidden, and output layers, respectively. The way to represent the needed weights is by creating two matrices: a 3x8 matrix for weights between the input and hidden layers and an 8x5 matrix for the weights between the hidden and output layers.

You need to store the weights and the network shape. It could be in an XML file or a user-defined binary file. If you were using python, you could save both matrices to a binary .npy file and encode the network shape in the file name. If you implemented the algorithm, you should store the persistent data. If, on the other hand, you are using an existing machine learning software package, it probably has its own I/O functions for storing and loading a trained network.

I hope this answer helps. And for more details, you can study the Neural Network Tutorial. Also, studying Machine Learning Algorithms will give you a generalistic approach to the same.

For an expert training on ML, you can join Intellipaat's Machine Learning Certification course.