Webpage scrolling in Selenium Python — how to scroll down and up, to element or with dynamic loading

Webpage scrolling in Selenium Python — how to scroll down and up, to element or with dynamic loading

When I am parsing and testing sites with Selenium, I often need to scroll the webpage, and with different conditions - up to a certain element or with dynamically loaded content. In this post, let's find out what scrolling methods there are and code examples for their use.

To execute this code, you must have Selenium and ChromeDriver installed on your computer. Selenium and Time must also be imported and a Selenium instance created.

To search for items on the page, you also need to import the By module. You also need to open the desired section of the site. In the code examples below I omit this, leaving only specific instructions.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://itwebmind.com/")

Scroll to the end

To make a page scroll using Selenium in Python, you can use running JavaScript code in the browser. Here's an example code snippet that will execute scrolling down.

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

JavaScript is executed using the execute_script() method. This method takes a JavaScript command as an argument.

Scroll to an element

If you want to scroll the viewport to a specific element, you can find that element using any of the available location methods (e.g., CSS selector, XPath) and then use the JavaScript scrollIntoView() method called by the execute_script() method.

element = driver.find_element(By.CSS_SELECTOR, "your-element-selector")
driver.execute_script("arguments[0].scrollIntoView(true);", element)

Note that we pass the element found in Selenium as the second argument of the execute_script() method, and in JavaScript code it is available as 0 element of the arguments array.

Scrolling dynamic pages

Some sites, such as social network feeds or video hosting sites, have dynamic content loading - new elements are added as you scroll.

If you use normal scrolling down, there will only be one content load, and there may be many more.

To avoid this error, we start scrolling in a loop - scroll to the end of the page, wait 2 seconds, check if the page height has changed, if yes, then scroll again.

last_height = driver.execute_script("return document.body.scrollHeight") 
while True: 
# Scroll down 
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 
# Pause while the page loads. 
sleep(2)
# Calculate the new scroll height and compare it to the last scroll height. 
new_height = driver.execute_script("return document.body.scrollHeight") 
if new_height == last_height: 
break 
print("Scrolling complete") 
last_height = new_height
print("New content has appeared, scroll further")

After scrolling to the bottom, the script "sleep" for 2 seconds. This is to give the browser time to load the new content on the page.

To run this code, you need to install the Selenium library and the ChromeDriver. Instructions on how to do this can be found on the Selenium website.

We've covered the main variants of scrolling implementation, I hope the article was useful. 

Please rate this article
(5 stars / 1 votes)