How to read a large file, line by line, in Python?

Question

I want to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

My code so far:

for each_line in fileinput.input(input_file):

    do_something(each_line)

    for each_line_again in fileinput.input(input_file):

        do_something(each_line_again)

Executing this code gives an error message: device active.

Any suggestions?

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line.

Answer 1

The best way to read a file in python is to open the file in a with block to avoid having to close it. Then, iterate over each line in the file object in a for loop and process those lines. e.g.:

with open("file.txt") as f:

    for line in f:

        #do your processing.

Answer 2

For reading a file line-by-line into a list you can use Input and Output:-

with open('filename') as f: 

lines = f.readlines()

or with stripping the newline character:

lines = [line.rstrip('\n') for line in open('filename')]

To know more about this you can have a look at the following video tutorial:-