Intellipaat Back

Explore Courses Blog Tutorials Interview Questions
+1 vote
2 views
in Python by (1.6k points)
I have a very big file 4GB and when I try to read it my computer hangs. So I want to read it piece by piece and after processing each piece store the processed piece into another file and read next piece.

Is there any method to yield these pieces ?

I would love to have a lazy method.

2 Answers

0 votes
by (25.1k points)

You can create a lazy method by using yeild:

def read_in_chunks(file_object, chunk_size=1024):

    """Lazy function (generator) to read a file piece by piece.

    Default chunk size: 1k."""

    while True:

        data = file_object.read(chunk_size)

        if not data:

            break

        yield data

f = open('really_big_file.dat')

for piece in read_in_chunks(f):

    process_data(piece)

0 votes
by (106k points)
edited by

The file.readlines() takes in an optional size argument which approximates the number of lines read in the lines returned.

bigfile = open('bigfilename','r')

tmp_lines = bigfile.readlines(BUF_SIZE)

while tmp_lines:

   process([line for line in tmp_lines])

   tmp_lines = bigfile.readlines(BUF_SIZE)

To know more about this you can have a look at the following video:-

Related questions

0 votes
1 answer
0 votes
1 answer
asked Jul 22, 2019 in Python by Vishal (106k points)
0 votes
1 answer
asked Oct 15, 2019 in Python by Sammy (47.6k points)
0 votes
1 answer

31k questions

32.8k answers

501 comments

693 users

Browse Categories

...