Back

Explore Courses Blog Tutorials Interview Questions
0 votes
1 view
in Python by (1.6k points)

I am writing a code to take an enormous textfile (several GB) N lines at a time, process that batch, and move onto the next N lines until I have completed the entire file. (I don't care if the last batch isn't the perfect size).

I have been reading about using itertools islice for this operation. I think I am halfway there:

from itertools import islice

N = 16

infile = open("my_very_large_text_file", "r")

lines_gen = islice(infile, N)

for lines in lines_gen:

     ...process my lines...

The trouble is that I would like to process the next batch of 16 lines, but I am missing something.

1 Answer

0 votes
by (25.1k points)

You can use the islice method in a loop to get the next n line from the file and perform your task if the method returns a None object that means that file has ended and you can quit the loop.

from itertools import islice

with open('filename.extension') as file:

    while True:

        next_n_lines = list(islice(file, n))

        if not next_n_lines:

            break

        # process next_n_lines

Welcome to Intellipaat Community. Get your technical queries answered by top developers!

28.4k questions

29.7k answers

500 comments

94.1k users

Browse Categories

...