Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in DevOps and Agile by (62.9k points)
closed by

I want to scrape all the data of a page implemented by an infinite scroll. The following python code works.

for i in range(100):

          driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

          time.sleep(5)

This means every time I scroll down to the bottom, I need to wait 5 seconds, which is generally enough for the page to finish loading the newly generated contents. But, this may not be time-efficient. The page may finish loading the new contents within 5 seconds. How can I detect whether the page finished loading the new contents every time I scroll down? If I can detect this, I can scroll down again to see more contents once I know the page finished loading. This is more time-efficient. 

closed with the note: repeated

Browse Categories

...