Screen Scraping a Javascript based webpage in Python

Question

I am working on a screen scraping tool in Python. But, as I look through the source of the webpage, I noticed that most of the data is coming through Javascript.

Any idea, how to scrape javascript based webpage ? Any tool in Python ?

Thanks

Why you do consume the Javascript directly ? For instance how do you call the JS function JS_Function(var1,var2,var3) from python ? — Kiran, CommentedNov 18, 2011 at 21:34

unutbu · Accepted Answer · 2011-11-18 14:09:22Z

5

Scraping javascript-based webpages is possible with selenium. In particular, try the Selenium WebDriver.

answered Nov 18, 2011 at 14:09

unutbu

882k196 gold badges1.8k silver badges1.7k bronze badges

I tried Selenium. I donot want to mimic the user action. As I see it from running a sample program, it opens browser window and mimics the action. I donot want that. I want to extract the data from the webpage into my code.
– Kiran
CommentedNov 19, 2011 at 11:18
1
You don't have to mimic user actions if you don't need to. Just download the page and parse it. The point of using selenium is that it processes javascript for you.
– unutbu
CommentedNov 19, 2011 at 12:28

Add a comment |

hoju · Accepted Answer · 2012-09-24 00:58:31Z

I use webkit, which is the browser renderer behind Chrome and Safari. There are Python bindings to webkit through Qt.

And here is a full Python example to execute JavaScript and extract the final HTML.

Will · Accepted Answer · 2011-11-18 14:10:56Z

3

You can use the QtWebKit module of the PyQt4 library

answered Nov 18, 2011 at 14:10

Will

4,5281 gold badge23 silver badges20 bronze badges

Add a comment |

Collectives™ on Stack Overflow

Screen Scraping a Javascript based webpage in Python

3 Answers 3

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Linked

Related