Python how to read N number of lines at a time

Question

I am writing a code to take an enormous textfile (several GB) N lines at a time, process that batch, and move onto the next N lines until I have completed the entire file. (I don't care if the last batch isn't the perfect size).

I have been reading about using itertools islice for this operation. I think I am halfway there:

from itertools import islice

N = 16

infile = open("my_very_large_text_file", "r")

lines_gen = islice(infile, N)

for lines in lines_gen:

     ...process my lines...

The trouble is that I would like to process the next batch of 16 lines, but I am missing something.

Answer 1

You can use the islice method in a loop to get the next n line from the file and perform your task if the method returns a None object that means that file has ended and you can quit the loop.

from itertools import islice

with open('filename.extension') as file:

    while True:

        next_n_lines = list(islice(file, n))

        if not next_n_lines:

            break

        # process next_n_lines