Error when extracting a JavaScript value from a variable with Selenium python

Question

I'm attempting to get the extract the value of the variable remainingTimeString on this website in python using Selenium webdriver. I'm trying to use the driver.execute_script() function. Here is my code:

import selenium.webdriver options = selenium.webdriver.FirefoxOptions() options.add_argument("--headless") driver = selenium.webdriver.Firefox(options=options) driver.get('https://shopgoodwill.com/item/151632327') print(driver.execute_script("return remainingTimeString")) 

However, when I run it, I get:

selenium.common.exceptions.JavascriptException: Message: ReferenceError: remainingTimeString is not defined 

What should I do? the variable is clearly in a script when I check the HTML source. Thanks!

Answer 1

That data is being pulled dynamically from an API, after page loads, so your options are - either use a WebDriverWait for that element (or an implicit wait for that matter), or use a less complex solution, like below (without selenium), where you inspect the Network Tab in Dev Tools, locate the API where data is being pulled from, and scrape that API directly:

import requests import pandas as pd headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" } url = 'https://buyerapi.shopgoodwill.com/api/ItemDetail/GetItemDetailModelByItemId/151632327' r = requests.get(url, headers=headers) df = pd.json_normalize(r.json()) print(df['remainingTime'][0]) 

Result printed in terminal:

'17h 56m ' 

The code above is extracting remainingTime only from that dataframe. There are detailed product infos in that json, so you can get other data as well, if you need. Python requests documentation can be found at https://requests.readthedocs.io/en/latest/

Also pandas documentation: https://pandas.pydata.org/pandas-docs/stable/index.html