Memory errors and list limits?

Question

I need to produce large and big (very) matrices (Markov chains) for scientific purposes. I perform calculus that I put in a list of 20301 elements (=one row of my matrix). I need all those data in memory to proceed next Markov step but I can store them elsewhere (eg file) if needed even if it will slow my Markov chain walk-through. My computer (scientific lab): Bi-xenon 6 cores/12threads each, 12GB memory, OS: win64

Traceback (most recent call last): 

 File "my_file.py", line 247, in <module> 

    ListTemp.append(calculus) 

MemoryError

Example of calculus results: 9.233747520008198e-102 (yes, it's over 1/9000)

The error is raised when storing the 19766th element:

ListTemp[19766] 

1.4509421012263216e-103

If I go further

Traceback (most recent call last): 

 File "<pyshell#21>", line 1, in <module> 

    ListTemp[19767] 

IndexError: list index out of range

So this list had a memory error at the 19767 loops.

Questions:

1. Is there a memory limit to a list? Is it a "by-list limit" or a "global-per-script limit"?

2. How to bypass those limits? Any possibilities in mind?

3. Will it help to use numpy, python64? What are the memory limits with them? What about other languages?

Answer 1

To get rid of the memory error and list limits you can create a binary file and check if the information is already in it if yes make a local variable to hold it else write some data you deem necessary.

Data = shelve.open('File01')

for i in range(0,100):

Matrix_Shelve = 'Matrix' + str(i)

if Matrix_Shelve in Data:

Matrix_local = Data[Matrix_Shelve]

else:

Data[Matrix_Selve] = 'somenthingforlater'

To know more about this you can have a look at the following video tutorial:-