I've been using a .NET framework to train a neural network, and my data set is really large, with a large number of inputs.
I've decided to switch to Encog due to some problems with the old framework I was using, and also because Encog seems much richer and offers additional features.
My problem is that Encog loads all training data at once, into a 2D array, and this doesn't suit me since I have a very large data set. I've tried working around this, and also checked other questions (such as this one but couldn't find a good answer).
I tried working with SQLNeuralDataSet and other implementations that support streaming operation but they all seem to eventually load the data into memory (using MemoryDataLoader) and this doesn't solve my problem.
Is there a way I can load (and train) my network one item at a time? Or is this option not available in Encog?
I ended up copying the source code and modifying it according to my needs. It all comes down to Process(IMLDataPair pair) in GradientWorker which, according to the documentation:
Process one training set element.