Invalid Selector Exception in Selenium with CSS selectors

Question

I am new to web crawling and I am trying to write a simple script to get course names from a University course catalog table:

from selenium import webdriver from selenium.webdriver.firefox.firefox_binary import FirefoxBinary binary = FirefoxBinary(r'C:\Program Files\Mozilla Firefox\firefox.exe') driver = webdriver.Firefox(firefox_binary=binary) url = 'https://courses.illinois.edu/schedule/2018/fall/CS' driver.get(url) course_names = [] for i in range(1, 69): if(float(i)%2 != 0): #odd row number curr_name = driver.find_element_by_css_selector('tr.odd:nth-child(i) > td:nth-child(2) > a:nth-child(1)').text else: curr_name = driver.find_element_by_css_selector('tr.even:nth-child(i) > td:nth-child(2) > a:nth-child(1)').text course_names.append(curr_name) print(course_names) driver.quit() 

When I run this I get the following error:

InvalidSelectorException: Message: Given css selector expression "tr.odd:nth-child(str(i)) > td:nth-child(2) > a:nth-child(1)" is invalid: InvalidSelectorError: 'tr.odd:nth-child(str(i)) > td:nth-child(2) > a:nth-child(1)' is not a valid selector: "tr.odd:nth-child(str(i)) > td:nth-child(2) > a:nth-child(1)" 

I am completely lost on how to get around this. I am just trying to get it to go through the table. It just does not seem to like i. I know this works:

tr.odd:nth-child(1) > td:nth-child(2) > a:nth-child(1) tr.even:nth-child(2) > td:nth-child(2) > a:nth-child(1) tr.odd:nth-child(3) > td:nth-child(2) > a:nth-child(1) 

Any suggestions?

Answer 1

There are multiple issues with your code:

• i is used as a character in your selector. Replace with nth-child(" + str(i) + ")

• you are filtering the odd and even rows in your script and in the selector. Choose one, not both.

• locating elements and reading the text in a loop is expensive. Scraping the text directly with some JavaScript would be a better approach.

rows = driver.execute_script(""" return [].map.call(document.querySelectorAll('#default-dt tbody tr'), row => [ row.cells[0].innerText, /* Course number */ row.cells[1].innerText, /* Course title */ row.querySelector('[href]').href /* Course link */ ]); """) for code, title, href in rows: print(code, title, href)