Selenium Webdriver using Proxy IP

Question

I was using selenium webdriver to scrap data from a website.due to heavy traffic from My IP, now i am not able to access website (may be my ip is blocked for the website).

Is there any way to setup the Proxy IP, so that it would be treated as a new IP every-time i run the webdriver..?

true it'll be treated as the new IP depending on the proxy IP pool you gonna use. — Igor Savinkin, CommentedOct 15, 2015 at 7:42
@IgorSavinkin but my concern is how to setup the Proxy ip with webdriver. — Pradeep Maurya, CommentedOct 15, 2015 at 7:44
I know it. So do you use Selenium with python? Could you expose what you've already done? — Igor Savinkin, CommentedOct 15, 2015 at 7:47
@IgorSavinkin i am using it with java. as well as just creating a FF driver instance using following code driver=new FirefoxDriver(); driver.get("whatsmyip.net/"); — Pradeep Maurya, CommentedOct 15, 2015 at 7:51

Helping Hands · Accepted Answer · 2015-10-15 08:11:40Z

You can set proxy IP with selenium webdriver as per following way :

FirefoxProfile profile = new FirefoxProfile(); profile.addAdditionalPreference("network.proxy.http", "localhost"); profile.addAdditionalPreference("network.proxy.http_port", "8080"); WebDriver driver = new FirefoxDriver(profile);

You should change your desired IP address and port in above code.

For more detail , Please have a look at : Selenium Webdriver with proxy

I have used above code & its working. Do you have any website where we can get the proxy IPs which will work 100% ,as i visited several sites (i.e proxylist.hidemyass.com/search-1291967#listable)they provide the ip & ports but by using them we are getting 50% success rate. — Pradeep Maurya, CommentedOct 15, 2015 at 9:09

Igor Savinkin · Accepted Answer · 2015-10-15 08:30:25Z

Since you scrape a lot you'll probably need a reliable proxy provider. So most of them provide their own api for auth and using their proxy pool. I've got this piece of code (java) from Luminati.io

package example; import org.apache.http.HttpHost; import org.apache.http.client.fluent.*; public class Example { public static void main(String[] args) throws Exception { HttpHost proxy = new HttpHost("zproxy.luminati.io", 22225); String res = Executor.newInstance() .auth(proxy, "lum-customer-CUSTOMER-zone-YOURZONE", "YOURPASS") .execute(Request.Get("http://www.telize.com/geoip").viaProxy(proxy)) .returnContent().asString(); System.out.println(res); } }

There are more complex example codes there.

Non-professional scrape

If you just want to plug in any deliberate proxy IP to test, you might use this:

FirefoxProfile profile = new FirefoxProfile(); host='149.215.113.110' port='9150' profile.SetPreference("network.proxy.type", 1); profile.SetPreference("network.proxy.http", host); profile.SetPreference("network.proxy.http_port", int(port)); driver = new FirefoxDriver(profile);

Collectives™ on Stack Overflow

Selenium Webdriver using Proxy IP

2 Answers 2

Non-professional scrape

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Non-professional scrape

Related