Lazy Method for Reading Big File in Python?

Question

I have a very big file 4GB and when I try to read it my computer hangs. So I want to read it piece by piece and after processing each piece store the processed piece into another file and read next piece.

Is there any method to yield these pieces ?

I would love to have a lazy method.

Answer 1

You can create a lazy method by using yeild:

def read_in_chunks(file_object, chunk_size=1024):

    """Lazy function (generator) to read a file piece by piece.

    Default chunk size: 1k."""

    while True:

        data = file_object.read(chunk_size)

        if not data:

            break

        yield data

f = open('really_big_file.dat')

for piece in read_in_chunks(f):

    process_data(piece)

Answer 2

The file.readlines() takes in an optional size argument which approximates the number of lines read in the lines returned.

bigfile = open('bigfilename','r')

tmp_lines = bigfile.readlines(BUF_SIZE)

while tmp_lines:

   process([line for line in tmp_lines])

   tmp_lines = bigfile.readlines(BUF_SIZE)

To know more about this you can have a look at the following video:-