Is there a way to train an Encog neural network without loading all training set in memory?

Question

I've been using a .NET framework to train a neural network, and my data set is really large, with a large number of inputs.

I've decided to switch to Encog due to some problems with the old framework I was using, and also because Encog seems much richer and offers additional features.

My problem is that Encog loads all training data at once, into a 2D array, and this doesn't suit me since I have a very large data set. I've tried working around this, and also checked other questions (such as this one but couldn't find a good answer).

I tried working with SQLNeuralDataSet and other implementations that support streaming operation but they all seem to eventually load the data into memory (using MemoryDataLoader) and this doesn't solve my problem.

Is there a way I can load (and train) my network one item at a time? Or is this option not available in Encog?

Thanks

EDIT

I ended up copying the source code and modifying it according to my needs. It all comes down to Process(IMLDataPair pair) in GradientWorker which, according to the documentation:

Process one training set element.

Answer 1

The is a source code for the data sources for Encog, there are several options. Look at Buffer/BufferedMLDataSet.cs, for example:

/// This class is not at all memory-based, so very long files can be used, without

/// running out of memory. This dataset uses an Encog binary training file as a

/// buffer.

What you have to do is to loop through your data set to build a CSV file with the inputs and outputs, then convert the CSV file (which was large) to binary using Encog Utility. Convert CSV Binary then load it into a trainingSet like this:

var trainingSet = new BufferedMLDataSet(binaryFile)

and pass the trainingSet to the BackPropagation object. It would be much better if you could load the training one by one.

If you wish to know more about Artificial Neural Network visit this Artificial Neural Network Tutorial.