How to select a dropdown menu using Selenium Python

Question

I'm working on a personal project, trying to use selenium to web scrape my locals teams results from a website. The site has three drop-down lists, at the moment I'd be happy to just be able to alter one. The code below clicks reject cookies when I enter the page, that's what the WebDriver line does. I'm getting as far as 'dropdown' and then an error that no such element exists. I've tried CSS_SELECTOR, XPATH and all the other options to no avail. I am only new to this so it could be something simple I'm missing but I've read a lot of the previous forums with similar questions and the answers didn't work for me.

from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.ui import Select import time driver = webdriver.Firefox() driver.get("https://www.leinsterrugby.ie/domestic-rugby/domestic-fixtures-results/") time.sleep(3) WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='onetrust-reject-all-handler']"))).click() time.sleep(3) dropdown=Select(driver.find_element(By.XPATH,value='//*[@id="groups_data"]') time.sleep(5) dropdown.select_by_value('5756') driver.quit() # close browser 

Answer 1

Many of those lines can be simplified by using SeleniumBase.

Here's code that searches for results on that site and then prints the results: (self.switch_to_frame() is needed to switch to the iframe, and self.select_option_by_text() is for the dropdowns.)

from seleniumbase import BaseCase class RugbyResultsTests(BaseCase): def test_get_rubgy_results(self): self.open("https://www.leinsterrugby.ie/domestic-rugby/domestic-fixtures-results/") self.switch_to_frame('iframe[src*="sportsmanager.ie/~leinsterrugby/"]') self.sleep(1) self.select_option_by_text("select#usercompyear", "2020-2021") self.sleep(3) self.select_option_by_text("select#groups_data", "Leinster Seconds League (J2)") self.sleep(1) self.click('a[href*="sportsmanager.ie/~leinsterrugby/league/155189"]') self.sleep(2) print("\n" + self.get_text("table.table-condensed")) 

Output: (After running the above test with pytest with seleniumbase installed via pip.)

POS TEAM PLD W D L PF PA DIFF BP BP L DED PTS 1 Ashbourne 1 1 0 0 18 17 1 0 0 0 4 2 Gorey 1 1 0 0 0 0 0 0 0 0 4 3 Kilkenny 1 0 0 1 17 18 -1 1 1 0 1 4 Suttonians 0 0 0 0 0 0 0 0 0 0 0 5 Monkstown 0 0 0 0 0 0 0 0 0 0 0 6 Seapoint 0 0 0 0 0 0 0 0 0 0 0 7 Bective Rangers 0 0 0 0 0 0 0 0 0 0 0 8 Dundalk 1 0 0 1 0 0 0 0 0 0 0 

The default browser for SeleniumBase tests is Chrome, but you can change that by calling pytest --firefox.

Answer 2

The element with dropdown options as Men, Schools, Women, Youth is within an <iframe> so you have to:

• Induce WebDriverWait for the desired frame to be available and switch to it.

• Induce WebDriverWait for the desired element to be clickable.

• You can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get('https://www.leinsterrugby.ie/domestic-rugby/domestic-fixtures-results/') WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src='https://shared2.sportsmanager.ie/~leinsterrugby/']"))) Select(WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select[name=user]")))).select_by_value('7183') 

  • Using XPATH:

    driver.get('https://www.t-online.de/themen/e-mail') WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@src='https://shared2.sportsmanager.ie/~leinsterrugby/']"))) Select(WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//select[@name='user']")))).select_by_value('7183') 

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC 

• Browser Snapshot: