How to scrape data generated by javascript using python

Question

I want to scrape the Number of participants of the following news. The url is http://news.sina.com.cn/c/2013-07-11/175827642839.shtml And I want to get the Number 820. It is generated by javascript. How can I get that number using simple way?

Answer 1

You could analize javascript code and do the same in python. Or you can use Selenium in Python.

edit:

Here example from selenium page changed to do what you need.

It open browser (firefox), wait 5 second (to load page) and get text

#!/usr/bin/python import selenium from selenium import webdriver from selenium.common.exceptions import NoSuchElementException from selenium.webdriver.common.keys import Keys import time browser = webdriver.Firefox() # Get local session of firefox browser.get("http://news.sina.com.cn/c/2013-07-11/175827642839.shtml ") # Load page time.sleep(5) # Let the page load try: element = browser.find_element_by_xpath("//span[contains(@class,'f_red')]") # get element on page print element.text # get element text except NoSuchElementException: assert 0, "can't find f_red" browser.close()