.writerow() csv in Python not writing all data

Question

I'm new to Python and I'm trying to scrape some data and save them in a csv. I'm trying to loop a csv with a list of URLs, read the data from each URL and write that information in another csv file

The following code is writing roughly half of the data in the cvs but is printing everything fine while it's writing

df_link = pd.read_csv('url_list')

with open('url_list.csv', newline='') as urls, open('output.csv', 'w', newline='') as output:

    csv_urls = csv.reader(urls)

    csv_output = csv.writer(output)

    csv_output.writerow(['details','date'])

    for link in df_link.iterrows():

        url = link[1]['url']

        browser.get(url)

        soup = BeautifulSoup(browser.page_source)

        csv_file = open('output.csv', 'w')

        csv_writer = csv.writer(csv_file)

        csv_writer.writerow(['details'])

        details=[i.text for i in soup.find_all(class_='product-info-content- 

        block product-info')]

        print('details :', details)

        dt = date.today()

        print('date :', dt)

        csv_output.writerow([str(details).strip('[]'), dt])

        csv_file.close()

Everything is being printed fine when the code is running, but not all the rows of data are being written in the output csv.

I hope someone can help. Thank you!

Answer 1

This issue is occurring because you are opening the csv file twice. The second time is in the loop with the write flag because of which the file gets overwritten on each iteration .

You should move that code out of the loop. Like this.

csv_file = open('output.csv', 'w')

csv_writer = csv.writer(csv_file)

csv_writer.writerow(['details'])